TII Infrared Tracking Challenge

The newest generation of autonomous vehicles are expected to successfully navigate in environments with challenging lighting conditions. This is where common, photometric visual perception-based tracking algorithms are put to their limits. We are now exploring the use of radiometric sensors, providing infrared frames (instead of photometric, RGB ones) for object tracking in low-light conditions. The challenge put forward is to develop approaches that can effectively track objects in the dark, in both structured and unstructured environments, including pedestrians, vehicles (cars, trucks, buggies, motorcycles), among others.

Challenge Teams are encouraged to leverage publicly available datasets with similar content to TII's labeled sample dataset and to utilize color-to-thermal domain adaptation techniques due to a lack of publicly available annotated thermal video sequences. Submissions will be evaluated against criteria set by TII, based on common tracking metrics in the RGB (red, green, blue) domain.

Note: further information on how to format your solution output has been provided in the Challenge Resources section: Click Here

Challenge Breakthrough

Develop an innovative algorithmic model to track objects in thermal images that will aid in decision-making. Submit your data to be scored against the challenge metrics and You could support the advancement of robotics and autonomous driving.

Background

In autonomous driving, a vehicle is equipped with sensors and uses deep-learning-based algorithms to process raw sensor information to understand the vehicle’s surrounding environment, detect and classify objects of interest, and predict their movement. An autonomous vehicle must have the capability to navigate in both high- and low-light conditions. However, difficulties arise when sensor information becomes limited due to low-light conditions. Consequently, standard RGB (red, blue, green) imaging is insufficient, resulting in a detection failure. One solution for this is to depend on a thermal (infrared) imaging sensor that captures heat instead of light.

Infrared (IR) tracking is a technique that uses infrared sensors to detect the presence and location of objects. In the context of autonomous driving, IR sensors can be used to detect obstacles in the vehicle’s environment, such as other vehicles or pedestrians. While there are several types of infrared sensors that can be used for tracking in autonomous driving, this challenge uses Boson infrared sensors and provides infrared frames for machine learning analysis and tracking. Active infrared sensors emit their own infrared radiation and use the reflections from that radiation to detect objects. Active sensors can be used to detect objects at much greater distances than passive IR sensors and can also be used to measure the distance to an object.

The Autonomous Robotics Research Center (“ARRC”), a research center within TII, has been established to investigate and advance a new era of robotics and autonomy.  ARRC has established a leadership position in its field, locally and globally, with in-depth technical expertise in robotics, computer vision, and bio-inspired technologies. ARRC is a UAE-based research center that aims to lead global advances in artificial intelligence, autonomous robotics, quantum computing, cryptography and quantum communications, directed energy, secure communication, smart devices, advanced materials, and propulsion and space technologies. In this Challenge, ARRC is making available to Challenge Teams about 5 GB of infrared imaging data for processing, which is available in the Resources tab and in a ROS bag format.

Robot Operating System (“ROS”) is a flexible framework for writing robot software. One of the tools that ROS provides is ROS bag, which is a tool for recording and playing back sensor data, including data from cameras. When using ROS bag to record sensor data, the sensor data is recorded to a file in a specific format (the “bag” format), which can be played back on either the same machine or a different machine, using the ‘rosbag’ command-line tool or a ROS-based application.

The goal of the Challenge is for each Challenge Team to develop  object detecting and tracking algorithms that will eventually be used to help create the navigation system on heterogenous environments in order to detect objects by thermal imaging, in particular scanning scenes with thermal imaging to identify human bodies.

The Challenge

Develop a neural network-based object classification model using the provided annotated thermal images that detects and tracks multiple instances, within the given set of object classes, at a time. 

     The winning Solutions will be able to:

Most Importantly (solutions should attempt to achieve these at a minimum. This is the core goal of the challenge):

• Track multiple objects in a variety of heterogeneous environments, based on thermal images,
• Robustly track multiple objects at the same time; and
• Run standalone and with few dependencies,

Bonus 1 (a small amount of additional points will be awarded to Solutions that achieve these):

• Precisely re-identify an object that has left and re-entered the field of vision, based on thermal images,
• Correctly classify between people and types of vehicles (e.g., cars, trucks, all-terrain vehicles (ATV), and buggies),
• Accurately detect objects of interests from thermal images, including those partially occluded

Bonus 2 (an even smaller amount of additional points will be awarded to Solutions that achieve this)

• Be run on an NVIDIA Jetson AGX Xavier board or similar at a high FPS rate, ideally at least 15 FPS (frames-per-second),

(“Objectives”).

The objectives stated above are separated by the way Solutions will be scored. On Your submission You will indicate which of these Objectives your Solution meets. The use of state-of-the-art domain adaptation techniques is encouraged and bonus points will be awarded for successful object re-identification. While developing Your Solutions, You should evaluate Your work against a developed metric. An example of a publicly available metric can be found at https://github.com/JonathonLuiten/TrackEval. The re-identification metric to be used in judging, described in the Judging Criteria section below, has a threshold for considering a match of 50% (i.e., mAP (mean Average Precision) @0.5).

The Data Provided

TII has provided data for the Challenge Teams to use, available in the Resources tab of this challenge page. The sample and test datasets will be provided in ROSBAG format as well as in RGB and thermal frames. Images were captured on a UGV vehicle with a maximum speed of 1.5 m/s. There are a total of 14,379 images available (5,784 daytime (provided in RGB and Infrared, and 8,595 nighttime (provided in Infrared only)). Specific information is available in the Readme file and Guidelines Update which are available in the data download and Resources tab, respectively.

Challenge Teams are encouraged to parse the data into their own training and validation sets to independently check their algorithms. Challenge Teams may use open source datasets. These datasets should be described in the submission form where a specific field is provided to capture them.

How do I Win?

Challenge Teams should pay particular attention to the judging criteria and minimum requirements detailed in these Guidelines. The evaluation of submitted algorithms will be based on metrics common to the fields of infrared imaging, robotic automation and object classification and tracking (e.g., multi-object tracking, precision, object reidentification and inference time).

Submitted Solutions should:

• Accept the ROS bag file format as an input (optional)
• Be provided as a dockerized package or binary with source code
• Output data in the required output format (.json file containing bbox and ID predictions per image frame, see the guidelines update for more information) - 2 new sample examples of output formats have been provided in the Resources tab

Submission Requirements

The submission deadline for Solutions in response to the Challenge is April 5, 2023 at 05:00 PM ET (New York). 

Submissions will consist of:

• A submission title, Challenge Team name, and the name of the individual who is the Challenge Team lead (“Challenge Team Leader”),
• A list of Challenge Team Members, their institutional/organizational affiliations and their role,
• Your submission file (dockerized or binary + a readme file) - file will be attached to the submission form. The file should be in .zip format;
• Submission files can be sent as a URL instead if hosted in cloud storage (e.g., OneDrive, Google Drive); and
• Important - Your submission file must include a readme file with build and run instructions, dependencies, the specific folder format for image storage (if applicable), and any flags that are required (if any). If TII is unable to run the submission file due to lack of clear instructions, we unfortunately cannot reach out for clarification,
• Write-up on Your solution. The narrative You provide shall include the following;
• A description of Your approach to solving the problem,
• A description of the neural network developed (if applicable),
• A description of Your computational platform including the hardware on which Your Solution was developed and tested and the GPUs You used for processing including RAM (8GB, 16GB, etc.),
• Descriptions and sources of any open-sources data used to aid in development/testing; and
• How the submission can be embedded on an AI board and the inference time it is capable of,
• Indicators on whether Your Solution meets the requested metrics (See above for: Most Important, Bonus 1, and Bonus 2). - Please note that Your final score will not be based strictly on whether you can accomplish all these fields or not. But by indicating whether Your solution meets them, Your submission will be measured against the relevant metrics. Answering “No” to any metric means Your submission may not be checked against that metric.

Judging Criteria

Judging of Solutions will occur in multiple stages: 

Stage 1 will apply a curve will to all submissions based on the submission with the highest HOTA score (HOTA: a Higher Order metric for evaluating multi-object Tracking). The top Solution submission’s score will determine the curve against which the other Solutions are scored. Your performance in this stage will be worth an estimated 80% of your overall submission score, so this is what your solution should primarily address.

Stage 2 will apply a reid criterion (instance re-identification) to the submissions. Your performance in this stage will be worth an estimated 15% of your overall submission score.

Stage 3 solutions will be installed on the AI embedded board and evaluated for compatibility and inference time. Your performance in this stage will be worth an estimated 5% of your overall submission score.

To summarize, in order to be eligible for a prize, Your Solution submission must, at minimum:

• Satisfy the Judging Criteria requirements.
• Thoughtfully address the Submission Form questions.
• Be scored higher than Your competitors!

Prize

Three winners will be awarded prizes totaling US $75,000 by May 17, 2023.

Timeline

Submission Form

Rules

The TII Infrared Tracking Challenge (“Challenge”) invites innovators, start-ups, research institutes and students (“Challenge Teams”, “Challenge Team Members”, “You” and “Your”) to enter. The solution to the Challenge provided by each Challenge Team referred to as, for each Challenge Team, the “Solution”.

The Challenge has been organized by Technology Innovation Institute – Sole Proprietorship LLC, a limited liability company registered pursuant to the laws of the Emirate of Abu Dhabi, United Arab Emirates, and having its registered office at Accelerator 2 Building, on Plot M12, Masdar City, Abu Dhabi, the United Arab Emirates – (“TII”, “We” or “Us”), the dedicated technology programme management pillar of the Advanced Technology Research Council (ATRC), the overarching advanced technology research body in Abu Dhabi.

The Challenge is governed by:

• the terms and conditions described in these TII Infrared Tracking Challenge Guidelines (the “Guidelines”);
• any further rules and regulations that are released to You in respect of the Challenge from time to time, and; 
• any documents shared with You during the course of the Challenge, relating to Your participation in the Challenge or otherwise.

The Challenge Teams are required to register to compete in the Challenge by clicking the "Solve this Challenge" button on the Challenge Page.  Challenge Teams will be asked to provide the following:

ELIGIBILITY AND CONFLICT OF INTEREST

Any Challenge Team that has the skills, resources and knowledge to participate and succeed in completing the Challenge is eligible, except:

• employees of TII, its parent company or other subsidiaries of such parent company (the “Affiliates”);
• employees of agents or suppliers of TII or any of its Affiliates, who are professionally connected with the Challenge or its administration; 
• members of the immediate families or households of (a) or (b); 
• any person or entity involved, or suspected of being involved, in illegal activities; 
• any person or entity registered or ordinarily resident in a country that is on a sanctions list at any time during this Challenge (including, but not limited to, the Sanctions Lists maintained by the United States, the United Nations and the European Union);
• any Challenge Team that does not hold the appropriate insurance requirements; or
• any other entity or person that TII may deem ineligible for whatsoever reason.
• If a Challenge Team becomes aware during the Challenge that it is, or will become, ineligible for the Challenge, or in the event of a conflict of interest, or in the event of a material breach of the Guidelines, it shall notify TII without delay. In TII’s sole discretion, any Challenge Team which is, or will become, ineligible or conflicted or in material breach of the Guidelines may be disqualified from the Challenge. 

JUDGING PANEL & FINAL DECISION 

The Challenge Solutions will be judged by a designated panel (“Judging Panel”). The Judging Panel is selected at the sole discretion of TII. Unless otherwise updated during the Challenge, the Judging Panel will comprise of the following individual members:

The Judging Panel shall have sole and absolute discretion to determine the methodology used by it to render its decisions, to determine the winners of the Challenge and to award any prize in connection with the Challenge. The decision of the Judging Panel will be final. The Challenge Teams and the Challenge Team Members agree not to dispute any decision of the Judging Panel.

TII reserves the right to withhold any payments of the prize money in the event the Challenge Team and/ or Challenge Team Members are in material breach of the Challenge, the Guidelines and/ or the HeroX T&Cs. TII has the right to disqualify the Challenge Team or any Challenge Team Members under the Guidelines, and/or it is determined that submissions did not sufficiently meet the assessment criteria and goals of the Challenge, Submission Requirements and/ or the Objectives. 

INTELLECTUAL PROPERTY

Challenge Team Background IP: means any Intellectual Property Rights that are owned by any or all of the Challenge Team and that were in existence prior to the commencement of the Challenge;

Foreground IP: means any Intellectual Property Rights created or arising pursuant to the Challenge from Challenge Team and/or any third parties acting on its behalf;

Intellectual Property Rights : means all patents, rights to inventions, copyright and related rights, rights in designs, database rights, source code, rights to use, and protect the confidentiality of, confidential information (including know-how and trade secrets) and all other intellectual property rights, in each case whether registered or unregistered and including all applications and rights to apply for and be granted, renewals or extensions of, and rights to claim priority from, such rights and all similar or equivalent rights or forms of protection which subsist or will subsist now or in the future in any part of the world, but for the avoidance of doubt, excluding any trade marks, business names and domain names, goodwill and/or the right to sue for passing off or unfair competition;

Team Technology / Team Technologies: means the Solution submitted by Challenge Team and used to compete in the Challenge, including all technology, software and other Intellectual Property Rights that form part of such Solution; 

Team Technology Licences : means the licenses to the Challenge Team Background IP, the Third Party IP and the Foreground IP;

Third Party IP: means all third-party technologies, methods, resources and Intellectual Property Rights that form part of the Team Technologies, or that are required to give effect to the Team Technologies, but not including Challenge Team Background IP or Foreground IP; and

TII Background IP: means all information and documentation developed by TII and/or any of its Affiliates or third parties acting on its behalf in relation to the Challenge (including the proprietary data set, upon which the Challenge is based, as uploaded to the HeroX platform, the Guidelines and any measurements, scoring, statistical and any other data collected or developed).

The Challenge Team acknowledges that all Intellectual Property Rights in the TII Background IP is owned by, and remains the property of, TII or its Affiliates, respectively, and the Challenge Team shall have no rights in or to such TII Background IP, other than as expressly set out in these Guidelines.

TII acknowledges that all Intellectual Property Rights in the Challenge Team Background IP is owned by the Challenge Team and/or the relevant third parties (as applicable), and TII shall have no rights in or to the Team Technologies, other than as expressly set out in these Guidelines. 

At the date of submission of the Solution, the Challenge Team shall provide TII with the details of all Intellectual Property Rights that form part of the Team Technologies by listing all of the Challenge Team Background IP and Third Party IP that form part of (or are intended to form part of) the Team Technology. The Challenge Team will also provide any additional information as TII may reasonably request. 

TII grants the Challenge Team a non-exclusive, royalty-free licence to use the Intellectual Property Rights in the TII Background IP for the purpose of competing in the Challenge for the duration of the Challenge.

Challenge Team grants TII a non-exclusive, perpetual, royalty-free and sub-licensable licence to use any Intellectual Property Rights in the Team Technology.

Challenge Team agrees that, if it is announced as the winner of the Challenge, it will automatically grant to TII and its Affiliates a fully-paid up, non-exclusive, sub-licensable, perpetual, royalty-free, worldwide licence to use, commercialise, copy and/or modify and exploit for any and all usage purposes the Intellectual Property Rights in the Team Technology. 

Challenge Team and TII shall do all such things and execute all such documents necessary in order to give effect to these Guidelines with respect to Intellectual Property Rights, without undue delay, and in any event within two (2) months of TII notifying Challenge Team that it is the winner of the Challenge. 

For six months after the conclusion of the Challenge, each Challenge Team shall grant TII a first right of refusal to negotiate the purchase of an exclusive licence over the Intellectual Property Rights in the Solution and Team Technology. 

Challenge Team will not use TII’s logos, brand names, slogans or other trademarks, or post or otherwise use any of TII’s confidential or proprietary information without its prior written permission.

If any claim is made (or is likely to be made) against TII and/or its Affiliates, that the Solution and/ or Team Technologies infringe any third party’s intellectual property, Challenge Team will use all reasonable efforts, at TII’s sole option and at Challenge Team’s sole expense, to (i) procure for TII and/or its Affiliates (as applicable) the right to continue to use the Challenge Team Solution (or any part thereof) in accordance with the terms of these Guidelines; or (ii) modify or replace the relevant part of the Challenge Team Solution so that it ceases to be infringing, provided that if Challenge Team modifies or replaces the Challenge Team Solution, the modified or replacement Challenge Team Solution must comply with the warranties contained in these Guidelines and TII shall have the same rights in respect thereof. 

Challenge Team shall not enter into any agreements that may conflict with these Guidelines, or that may otherwise either restrict or affect TII's ability to enforce its rights under these Guidelines, including in relation to its use of the Challenge Team Solution licence.

EXPORT CONTROL 

In the event an export control application is required for any Solution, TII will make the application at its own cost and the Challenge Team will take all reasonable steps necessary to support TII in that application at its own cost. Challenge Team shall comply with all export control laws as may be applicable to it from time to time.

CHALLENGE TEAM RESPONSIBILITY AND FUNDING

Each Challenge Team is responsible for the acts or omissions of each of its Challenge Team Members during its participation in the Challenge, including compliance with these Guidelines.

Challenge Team must comply with all laws and regulations that apply to their participation in the Challenge, including laws relating to anti-bribery and anti-corruption.

All costs of competing in the Challenge are the responsibility of the competing Challenge Team and TII shall not be liable for any costs incurred by the Challenge Team during its participation in the Challenge. 

If a Challenge Team wishes to receive any third-party funding towards its participation in the Challenge, the Challenge Team should obtain written consent from TII prior to obtaining any such funding.

CHALLENGE TEAM CHANGES

After the registrations, Challenge Team may not, at any time throughout the Challenge, add or remove Challenge Team Members or merge with another Challenge Team, unless otherwise agreed in advance in writing with TII. 

SOCIAL MEDIA USE

Both during and after the Challenge, Challenge Team may: 

• post on social media or otherwise publicize that it is an entrant to or or winner of the Challenge.  To the extent Challenge Team does post on social media or otherwise publicize in connection with the Challenge, it shall make clear that it is speaking in its own capacity and not on behalf of TII; 
• not do anything to jeopardise the potential Intellectual Property Rights in its Solution through the use of social media or otherwise;
• not post disparaging or defamatory statements about TII, any Affiliate, partner, sponsor or anyone working with TII on this Challenge, the Challenge itself, or post anything TII, in its sole discretion, would find offensive, including discriminatory comments, insults or obscenity; 
• if it discloses its affiliation with the Challenge following its termination or expiry, clearly state that its views do not represent those of TII; and
• if it issues a publication, whether on social media or otherwise, describing its Solution, will describe the data set to which the Challenge Team Solution was applied in the Challenge and will expressly state that the data set is the property of TII.

PUBLICITY

TII has the right to publish details of all of the Solutions submitted by the Challenge Teams. In the event of a publication, it will notify the Challenge Team. 

No Challenge Team may publish details of its Solution without TII’s prior written consent. In the event of any such publication, TII will notify the Challenge Team whether it wishes to be named in any publication by the Challenge Team and the Challenge Team shall comply with TII’s request.  

LIMITATION OF LIABILITY

Insofar as is permitted by law, TII, its agents or distributors will not in any circumstances be responsible or liable to compensate a Challenge Team for any loss or damage incurred by a Challenge Team, as a result of taking part in the Challenge, except where such loss or damage is directly caused by the negligence of TII, its agents or distributors or that of its employees.  Where TII is liable to compensate a Challenge Team for such loss or damage then, insofar as is permitted by law, such loss of damage shall be limited to an aggregate total across all events giving rise to such liability of $75,000.   

DATA PROTECTION 

TII will only process Challenge Team’s personal information as set out in the Privacy Policy posted to the Challenge Resources section and TII shall comply with its applicable obligations under data protection and privacy laws in relation to its collection and use of the Challenge Team Member’s personal information.

CONFIDENTIALITY

Subject to the TII’s right to publish details of all of the Solutions submitted by the Challenge Teams (as set out in the Publicity section above), all information identified as proprietary and submitted by Challenge Team to TII, or by TII to Challenge Team, relating to the Challenge will remain strictly confidential, unless a) otherwise agreed by Challenge Team and TII; b) as may be required by law; c) as may be required in order to carry out obligations under the Guidelines; and/or d) as set out in the Guidelines. 

The Challenge Team and TII agree to hold information received from each other in connection with the Challenge in strict confidence and that each will not disclose, copy, reproduce or distribute any of it without the consent of the provider of the information, during the Challenge and for a period of 2 years after the Challenge. The Challenge Team and TII retain all rights in its confidential information. 

TERM AND TERMINATION

TII shall not be liable for any failure in completion of the Challenge or delay in the performance of the Challenge caused by an event beyond its control.  

The Guidelines shall apply to Challenge Team from the Effective Date and, unless terminated earlier, shall end upon the winner of the Challenge being selected, save in respect of terms which are of an ongoing nature (such as ongoing licences or the right to publish).

If there is any reason to believe that a Challenge Team has breached the Guidelines or should no longer participate in the Challenge, including for breach of the Guidelines, conflict or ineligibility, TII may, at its sole discretion, disqualify the Challenge Team from participating in the Challenge and terminate these Guidelines with the Challenge Team. Upon such termination, each party will return the other party’s confidential information, save as otherwise set out in the Guidelines.

TII also reserves the right to terminate, hold void, suspend, cancel or amend the Challenge where it becomes necessary or desirable (in TII’s sole opinion) to do so.

Whether during the Challenge or once a Challenge Team has withdrawn or is otherwise disqualified from the Challenge or after the conclusion of the Challenge, the Challenge Team shall not engage in any conduct that is determined by TII to reflect poorly on or be disparaging to TII or and/or any Affiliate or that may disrupt or harm, in any manner, the Challenge. 

INSURANCE

For the duration of the Challenge and for six (6) years thereafter, Challenge Team shall have and maintain in place sufficient insurances with a reputable company covering, as a minimum, death, personal injury and damage to property caused by Challenge Team arising from Challenge Team’s involvement in the Challenge. TII reserves the right to request evidence of such insurances. 

SUBCONTRACTING AND ASSIGNMENT 

Challenge Team shall not sub-contract the performance of any of its obligations under the Guidelines without the prior written consent of TII. Registration in the Challenge is also non-transferable and so Challenge Team shall not assign or transfer such registration or any of its rights under the Guidelines without TII’s prior written approval.

TII may assign, delegate or transfer any of its rights or interests or duties under the Guidelines at its absolute and sole discretion.

Challenge Team and TII shall be responsible for all acts and omissions of its sub-contractors and the acts and omissions of those employed or engaged by the sub-contractors as if such acts and omissions were its own.

DISPUTE RESOLUTION

If a dispute or claim arises out of or in connection to the Guidelines (a “Dispute”), then either party may give to the other notice in writing as to such Dispute, including supporting documents. Upon receipt of such notice, TII and Challenge Team shall attempt in good faith to resolve the Dispute. If not resolved within 30 days of service of the dispute notice, the CEO or equivalent of each party shall use their reasonable endeavours to resolve such Dispute in good faith. If not resolved within a further 30 days, either party may refer the Dispute for determination by one arbitrator in London under the London Court of International Arbitration (LCIA) Rules, whose decision will be final and binding on the parties.

GENERAL

These Guidelines and any dispute or claim (including non-contractual disputes or claims) arising out of or in connection with it or its subject matter or formation shall be governed by and construed in accordance with the laws of the Emirate of Abu Dhabi.  The official language of the Challenge shall be English.  Any communication and notices between TII and Challenge Team shall occur via Challenge Team Lead, and shall be in writing, and may be by electronic mail. 

TII and Challenge Team shall, at the request and cost of the other, use all reasonable endeavours to promptly do or procure the doing of all such further acts, and execute and deliver or procure the valid execution and delivery of all such documents, as may from time to time be necessary to give full effect to the Guidelines and to secure the full benefit of the rights, remedies and benefits conferred on it by the Guidelines.

The Guidelines may be executed in counterparts, and together executed and delivered counterparts shall constitute a complete, binding contract. Facsimile or electronic execution shall have the same weight and effect as originals.

Waiver of a breach of any term of the Guidelines shall not operate as a waiver of breach of any other term or any subsequent breach of that term.

TII may at any time, without notice to Challenge Team, set off any liability of Challenge Team to it against any liability it has to Challenge Team, whether either liability is present or future, liquidated or unliquidated, and whether or not either liability arises under the Guidelines. Any exercise by TII of its rights under this sub-section shall not limit or affect any other rights or remedies available to it under the Guidelines or otherwise.

Additional Information

• By participating in the Challenge, each Challenge Team Member agrees to adhere to the HeroX Intellectual Integrity Policy and promises to submit only their original idea. Any indication of "copying" amongst competitors is grounds for disqualification.
• All applications will go through a process of due diligence; any application found to be misrepresentative, plagiarized, or sharing an idea that is not their own will be automatically disqualified.
• All ineligible applicants will be automatically removed from the Challenge with no recourse or reimbursement.
• No purchase or payment of any kind is necessary to enter or win the competition.
• Void wherever restricted or prohibited by law.