Validations UI
For OP and QC
Audit Node
While Publishing the Taskflow
Creating a New Taskflow
Begin by navigating to the Taskflow section in Quickserve.
Click on the Create New Taskflow button, which will prompt you to input the required details for the taskflow.
Fill in the necessary information like taskflow name, description, and task settings.
Attaching a Validation Script
During the taskflow creation, you will find an option labeled Validation Script. This is where you insert your validation logic.
You can use the appropriate decorators and function structures to specify the type of validation, such as frame-wise, class-wise, or task attribute validation (discussed in the next section).
Save
After adding the validation script, click the Save button to store the taskflow.
Note: In case of any syntactical error, the platform will identify it and ask the user for resolution.
After Publishing the Taskflow
Accessing the Published Taskflow
Once a taskflow is published, you can still make changes to the validation section by navigating to the Taskflow section.
Locate the published taskflow from the list of existing taskflows and click on it to open the detailed view.
Editing the Validation Script
In the detailed view of the published taskflow, there is an option called Validation Script.
This allows you to modify the validation logic post-publication. You can make updates to the script to adjust validation parameters, add new checks, or update the logic to reflect changes in the task.
Saving Changes
After making the necessary edits, click Save. The updated validation script will take effect immediately. In case of any syntactical error, the platform will identify it and ask the user for resolution.
Important Note:
In case of any syntactical error, the platform will identify it and ask the user for resolution.
Post-publication edits to the validation script will take affect to batches where the user will explicitly trigger the ‘Validation' feature again. This is applicable to OP, QC and Audit Nodes by triggering the ‘Validation’ feature with the help of the below buttons
OP and QC node users will be able to trigger the validation script from the tool, by clicking on either options.
Audit node users will have the script executed after each QC submission. When the audit node user will open the sequence (after the script is executed) in the Audit tool, the report will be available. If the user wants to execute one more time, they will open the report and click 'Rerun'.
Version Management After Publishing
Tracking Revisions
Quickserve tracks every change made to the validation script after the taskflow is published. Each modification is stored as a revision, which can be viewed in the Revision History section of the taskflow.
Accessing Revision History:
You can access the revision history from the taskflow page. This log will display all past versions of the validation script, including the date and details of the changes made.
Restoring Previous Versions
If needed, you can revert to any previous version of the validation script by selecting it from the revision history and clicking Restore.
This feature is useful for situations where a newly published script introduces issues or errors, allowing you to quickly roll back to a stable version without manual re-entry.
Example:
Version 1.0: Initial script with frame-wise validation
Version 1.1: Updated script with added class-wise validation
If errors are found in Version 1.1, you can restore Version 1.0 for immediate use while debugging the newer script. So restoring previous version,we have to click on Update and it is available along with new version.
Types of Validation Scripts
In Quickserve, validation scripts are executed using a decorator-based approach. Each type of validation uses specific decorators to target different elements like frames, classes, or task attributes. Here, we explain the types of validation, the usage of decorators, and how parameters in functions are handled.
1. Frame-wise Validation
Frame-wise validation ensures that each individual frame is validated according to specific rules. A decorator for this type of validation specifies that the function will process one frame at a time.
Decorator Usage:
The decorator
@dec({"type": "frame"})tells the system that the function is working on individual frame data.
Function Definition:
The function receives one parameter, which is the data for a specific frame.
The function processes this frame and can return an error log if any issue is found.
Explanation:
@dec({"type": "frame"})– This tells the function that it will receive data for a single frame.frame_data– This parameter holds the data for the particular frame from which ‘Validation’ feature was triggered, which can be processed as needed.For OP/QC - The function checks the data, and if the validation fails, it adds an error to the report.
Prior to launching Audit batch - The function checks the data, and if the validation fails, it adds an error to the log.
Example Function:
@dec({"type": "frame"}) def frame_wise_function(frame_data): print("Validating frame data") print(frame_data) error_log = [] # Check condition and log error if condition is met temp_err = { "type": "Error", "message": "<error message>", "object_type": "object_type", "frame_no": 1, # The current frame number "identity": "identity", "label": "label", "meta_info": {"<attribute/class key>":"<attribute/class value>"}, "blocker": False } error_log.append(temp_err) return error_log
Example Input
[ { "_id": "66e2f448e15e7b022090ff20", "seq_no": "1", "batch_id": "66e2f2c2dcb363000170da30", "key": "relationship_1", "idbkey": "66e2f2c2dcb363000170da30_1", "relationship": [ { "label": "isAttachedTo", "peers": [ { "objectClass": "7f23068c-b22a-46ca-9b1d-753380cba487", "subjectClasses": [ "d29b7069-a17d-47ee-9fb2-c0453cc8e77f" ] } ] } ], "updated_on": 1726149704033.0, "node_wise_batch_details_id": "66e2f2c2dcb363000170da30", "uniqueToken": "66e2f316e15e7b742f90fdc8" }, { "_id": "66e2f366e15e7b3c6190fe0f", "id": "00dd3faa-94c0-4ad2-ad32-b77ad60912b9", "identity": 1, "createdAt": "2024-09-12 13:57:03", "classId": "2", "class": "ShoppingCart", "color": "", "classColor": "#BEB304", "geometry": { "id": "737224c9-faec-4742-b413-e58659e368c4", "position": { "x": 24.59950017929077, "y": 10.930107539207684, "z": 1.061704987240236 }, "rotation": { "x": 0, "y": 0, "z": 0 }, "boxSize": { "x": 16.323409974480473, "y": 16.930311238828978, "z": 16.323409974480473 }, "clipTask": 1, "referenceSourceIds": [] }, "isVisible": true, "isLocked": false, "deleted": false, "discarded": false, "object_type": "cuboid", "taxonomy_attribute": { "isAttachedTo2D": "", "truncation": "0", "occlusion": "0", "isIgnoreArea": false, "isGroup": false }, "updated_on": 1726149478185.0, "order": 0, "verticalLimit": [ -14.3, 2.900000000000002 ], "groundClippingMode": 0, "isGeometryKeyFrame": true, "attributesKeyFrame": [], "isFirstTimeAttribute": false, "referenceIndex": -1, "keyFrame": "2_1_-1", "key": "1_00dd3faa-94c0-4ad2-ad32-b77ad60912b9", "idbkey": "66e2f2c2dcb363000170da30_1_00dd3faa-94c0-4ad2-ad32-b77ad60912b9", "seq_no": "1", "isUpdated": false, "node_wise_batch_details_id": "66e2f2c2dcb363000170da30", "uniqueToken": "66e2f316e15e7b742f90fdc8" }, { "_id": "66e3d576e15e7ba01191214b", "timeSpent": 193, "updated_on": 1726207350025.0, "seq_no": "1", "batch_id": "66e2f2c2dcb363000170da30", "key": "formInput_1", "idbkey": "66e2f2c2dcb363000170da30_1", "node_wise_batch_details_id": "66e2f2c2dcb363000170da30", "uniqueToken": "66e2f316e15e7b742f90fdc8", "taskAttribute": { "isCar": true, "isHuman": 1, "isTest": 1 }, "percentage": 41.7 }, { "_id": "66e2f34ee15e7bcdf890fdf3", "id": "f7789a37-58b8-4481-bbde-94f5c46032b4", "identity": 1, "createdAt": "2024-09-12 13:56:51", "classId": "1", "class": "Bicycle", "color": "", "classColor": "#e76fec", "geometry": { "id": "ce12eb07-0df8-49f3-9fbf-4a55802244aa", "position": { "x": -6.900500297546387, "y": 7.399999618530273, "z": 2.7559081065064275 }, "rotation": { "x": 0, "y": 0, "z": 0 }, "boxSize": { "x": 19.711816213012856, "y": 19.711816213012856, "z": 19.711816213012856 }, "clipTask": 1, "referenceSourceIds": [] }, "isVisible": true, "isLocked": false, "deleted": false, "discarded": false, "object_type": "cuboid", "taxonomy_attribute": { "isAttachedTo2D": "1", "occlusion": "0", "truncation": "0", "isIgnoreArea": false, "isGroup": false, "tilt": "90", "hasOversizedCargo": false }, "updated_on": 1726149454100.0, "order": 0, "verticalLimit": [ -7.1000000000000005, 2.8999999999999995 ], "groundClippingMode": 0, "isGeometryKeyFrame": true, "attributesKeyFrame": [ "isAttachedTo2D" ], "isFirstTimeAttribute": false, "referenceIndex": -1, "keyFrame": "1_1_-1", "key": "1_f7789a37-58b8-4481-bbde-94f5c46032b4", "idbkey": "66e2f2c2dcb363000170da30_1_f7789a37-58b8-4481-bbde-94f5c46032b4", "seq_no": "1", "isUpdated": false, "node_wise_batch_details_id": "66e2f2c2dcb363000170da30", "uniqueToken": "66e2f316e15e7b742f90fdc8" } ]
2. Class Validation Across Multiple Frames
Class validation checks specific classes across multiple frames. This ensures that particular classes (e.g., "Vehicle", "Trucks", etc.) are consistently annotated across all frames.
Decorator Usage:
The decorator
@dec({"type": "object", "selector": "class", "value": ["<class1>", "<class2>"]})tells the system to pass annotations of the specified classes across all frames into the function.
Function Definition:
The function receives one parameter for each specified class. If you validate two classes, you need two parameters.
The function compares or processes the classes across multiple frames.
Explanation:
@dec({"type": "object", "selector": "class", "value": ["<class1>", "<class2>"]})– This tells the system that the function should validate annotations for "<class1>" and "<class2>" across multiple frames.class1_dataandclass2_data– These parameters contain the annotations for the "class1" and "class2" classes, respectively, from all frames.The function processes these annotations, checking for errors across frames and logging issues accordingly.
Example Function:
@dec({"type": "object", "selector": "class", "value": ["Boar", "Camel"]}) def multiclass_check(boar_data, camel_data): print("Validating class1 and class1 across frames") print(class1_data) print(class2_data) error_log = [] # Check condition and log error if condition is met temp_err = { "type": "Error", "message": "<error message>", "object_type": "object_type", "frame_no": 1, "identity": "identity", "label": "label", "meta_info": {"<attribute/class key>":"<attribute/class value>"}, "blocker": False } error_log.append(temp_err) return error_log
Example Input
[ { "_id": "66e2f429e15e7b084990ff04", "id": "46c21f97-d769-4494-b7c4-3f4cd2be829e", "identity": 1, "createdAt": "2024-09-12 14:01:01", "classId": "7", "class": "Boar", "color": "", "classColor": "#5CF6F2", "geometry": { "id": "7f23068c-b22a-46ca-9b1d-753380cba487", "boxSize": { "x": 15.14778523545462, "y": 15.14778523545462, "z": 15.14778523545462 }, "position": { "x": 13.0989990234375, "y": -14.201000213623047, "z": 0.4738926177273095 }, "rotation": { "x": 0, "y": 0, "z": 0 }, "clipTask": 1, "referenceSourceIds": [] }, "isVisible": true, "isLocked": false, "deleted": false, "discarded": false, "object_type": "cuboid", "taxonomy_attribute": { "isAttachedTo2D": "", "occlusion": "0", "truncation": "0", "isIgnoreArea": false, "isGroup": false }, "updated_on": 1726149673378.0, "order": 0, "verticalLimit": [ -16.6, 2.900000000000002 ], "groundClippingMode": 0, "isGeometryKeyFrame": false, "attributesKeyFrame": [], "isFirstTimeAttribute": false, "referenceIndex": -1, "keyFrame": "7_1_-1", "key": "3_46c21f97-d769-4494-b7c4-3f4cd2be829e", "idbkey": "66e2f2c2dcb363000170da30_3_46c21f97-d769-4494-b7c4-3f4cd2be829e", "seq_no": "3", "isUpdated": false, "node_wise_batch_details_id": "66e2f2c2dcb363000170da30", "uniqueToken": "66e2f316e15e7b742f90fdc8" }, { "_id": "66e2f429e15e7bc86c90ff09", "id": "4693fcb6-7850-4a1f-99e1-b2d3b170f75f", "identity": 1, "createdAt": "2024-09-12 14:01:01", "classId": "7", "class": "Boar", "color": "", "classColor": "#5CF6F2", "geometry": { "id": "7f23068c-b22a-46ca-9b1d-753380cba487", "boxSize": { "x": 15.14778523545462, "y": 15.14778523545462, "z": 15.14778523545462 }, "position": { "x": 13.0989990234375, "y": -14.201000213623047, "z": 0.4738926177273095 }, "rotation": { "x": 0, "y": 0, "z": 0 }, "clipTask": 1, "referenceSourceIds": [] }, "isVisible": true, "isLocked": false, "deleted": false, "discarded": false, "object_type": "cuboid", "taxonomy_attribute": { "isAttachedTo2D": "", "occlusion": "0", "truncation": "0", "isIgnoreArea": false, "isGroup": false }, "updated_on": 1726149673379.0, "order": 0, "verticalLimit": [ -16.6, 2.900000000000002 ], "groundClippingMode": 0, "isGeometryKeyFrame": false, "attributesKeyFrame": [], "isFirstTimeAttribute": false, "referenceIndex": -1, "keyFrame": "7_1_-1", "key": "8_4693fcb6-7850-4a1f-99e1-b2d3b170f75f", "idbkey": "66e2f2c2dcb363000170da30_8_4693fcb6-7850-4a1f-99e1-b2d3b170f75f", "seq_no": "8", "isUpdated": false, "node_wise_batch_details_id": "66e2f2c2dcb363000170da30", "uniqueToken": "66e2f316e15e7b742f90fdc8" } ]
3. Task-wise Attribute Validation Across Frames
This type of validation ensures that attributes related to a task are validated across all frames in the task.
Decorator Usage:
The decorator
@dec({"type": "attribute", "selector": "task"})indicates that the function will work with task-level attributes across multiple frames.
Function Definition:
The function receives one parameter that includes all attributes related to the task, across all frames.
Explanation:
@dec({"type": "attribute", "selector": "task"})– This decorator targets task-level attributes across frames.task_data– The parameter contains task-related attribute data from all frames, which is processed for validation.The function logs any errors based on the validation criteria, returning the error log if issues are detected.
Example Function:
@dec({"type": "attribute", "selector": "task"}) def task_attribute_test(task_data): print("Validating task attributes across frames") print(task_data) error_log = [] # Check condition and log error if condition is met temp_err = { "type": "Error", "message": "<error message>", "object_type": "object_type", "frame_no": 1, "identity": "identity", "label": "label", "meta_info": {"<attribute/class key>":"<attribute/class value>"}, "blocker": False } error_log.append(temp_err) return error_log
Example Input
[ { "_id": "66e3d576e15e7b6eea91214c", "timeSpent": 0, "updated_on": 1726207350025.0, "seq_no": "2", "batch_id": "66e2f2c2dcb363000170da30", "key": "formInput_2", "idbkey": "66e2f2c2dcb363000170da30_2", "taskAttribute": { "isCar": true, "isHuman": 1, "isTest": 1 }, "percentage": 30.81, "node_wise_batch_details_id": "66e2f2c2dcb363000170da30", "uniqueToken": "66e2f316e15e7b742f90fdc8" }, { "_id": "66e3d576e15e7b5cfc912152", "timeSpent": 0, "updated_on": 1726207350025.0, "seq_no": "9", "batch_id": "66e2f2c2dcb363000170da30", "key": "formInput_9", "idbkey": "66e2f2c2dcb363000170da30_9", "node_wise_batch_details_id": "66e2f2c2dcb363000170da30", "uniqueToken": "66e2f316e15e7b742f90fdc8" }, { "_id": "66e3d8f3e15e7ba6d1912213", "timeSpent": 50, "updated_on": 1726208243260.0, "seq_no": "4", "batch_id": "66e2f2c2dcb363000170da30", "key": "formInput_4", "idbkey": "66e2f2c2dcb363000170da30_4", "node_wise_batch_details_id": "66e2f2c2dcb363000170da30", "uniqueToken": "66e2f316e15e7b742f90fdc8", "taskAttribute": { "isCar": false }, "percentage": 27.29 } ]
Error Handling and Return Format
Errors in Quickserve must be returned in a structured format that the system can interpret. The standard format includes fields for the error type, message, object type, frame number, identity, label, and any additional metadata.
Note: The error return format should follow the below format so that the tool can help visualize the error to the end user.
Error/Return Format Example:
{ "type": "Error", "message": "<error message>", "object_type": "object_type", "frame_no": 1, "identity": "identity", "label": "label", "meta_info": { "<attribute/class key>":"<attribute/class value>" }, "blocker": False }
type– Specifies that this is an error.message– A descriptive message about the error.object_type,identity,label– Contextual information about the object and annotation being validated.frame_no– The frame in which the error occurred.meta_info– Additional metadata, such as whether an annotation is attached to a 2D object.blocker– Indicates whether this error blocks further processing.
Packages and Versions
The following packages are available in the Quickserve validation environment along with python 3.9
Shapely v1.8.5 – for geometric operations
Scipy v1.9.3 – for scientific and technical computing
Numpy v1.24.0 – for numerical processing
For additional package support, contact the system administrator, as other packages are not supported by default.
Last updated