border border

AIFRED (AI Form Recognition Entity Detection)

Team members

Alex Wang Wei Jie (ISTD), Tan Jun Yong (ESD), Shang Zewen (ISTD), Jose Johnson Emerson Raja (ISTD), Zhang Jiazheng (ISTD), He Yuhang (ISTD)

Instructors:

Georgios Piliouras, Matthieu De Mari, Natarajan Karthik Balakrishnan

Writing Instructors:

Rashmi Kumar

Teaching Assistant:

Siddharth Kumar

 

 

What is AIFRED ?

 

AIFRED is an AI-powered mobile application that recognizes and extracts information from delivery documents from various aviation companies.

 

Why automate the manual process?

 

Collins Aerospace is a leader in the global aviation industry and is the world’s largest supplier of aerospace components. One of their operations in Singapore involves the repair and maintenance of customer’s airplane components in the region. This entails receiving and delivering hundreds of components on a daily basis. The logistics centre aids in processing all of these components through specialised physical documents that arrive along with the components. The information from these physical documents needs to be extracted into the internal system at Collins to track each component. Currently, this information extraction is done by keying in information manually via a computer, which makes it time-consuming and prone to administrative errors. The implications of an error can be disastrous since any incorrect information in the system will impede the maintenance process by incorrectly identifying the parts and components in use on an aircraft.

 

Why AIFRED

centered image

Manual data entry of a single document into the internal system takes more than 5 minutes. Comparatively, AIFRED is able to reduce the time taken to less than 1 minute, which is an 80% reduction.

 

Instead of detecting a document from an image one by one, AIFRED's mobile application supports batch processing. For a batch of 20 documents, the reduction in time is from 14s to 7s, which is a 50% reduction.

 

 

 

 

 

 

 

What are the key features of AIFRED ?

 

 (1) Identify and extract information from the forms provided

 (2) Data capture needs to be done on a smartphone for portability

 (3) AI & ML techniques are used to extract information and reduce human errors

 (4) Solution should be scalable, adding more types of forms for recognition

 (5) Solution should be integrated into Collin's existing infrastructure

 

 

 

 

 

 

 

 

Under the hood: AIFRED's AI

  1. The AI behind AIFRED contains three main models:
  2.  

(1) Document Classification Model - Used to classify the input data form.

AIFRED adopts state-of-the-art deep learning models to identify the form type at the pre-processing stage.

      

(2) Template-based Key-Value Extraction Model - Used to extract information from ARC forms.

 

Two main tasks are handled by AIFRED's Template-based Key-Value Extraction model

- Construct Table Structure: Takes advantage of the border information to construct a table structure 

- Template Mapping: Utilize document domain-specific knowledge such as a pre-defined table template to extract key-value pairs 

 

(3)  Deep-learning key-Value Extraction model - Used to extract information from PO and AWB forms.

 

Two main tasks are handled by AIFRED's Deep-Learning Key-Value Extraction model

- Entity Extraction: Employ state-of-the-art deep learning models and utilize text, position, visual information to extract Key and Value entities

- Key-Value Linking: Use deep learning models to study position information and link related key and value entities and form key-value pairs 

 

  1. Each model has been trained with thousands of real data to guarantee the desired performance.

 

 

 

 

 

 

 

AIFRED's front end mobile application

 

AIFRED's mobile application ensures that the client has an intuitive interface to collaborate with the strong AI backend model. It aids in automating the manual process of extracting information from physical forms.

 

The application is built with Flutter for its cross-platform compatibility, fast performance and strong support for machine learning libraries.

 

 

 

 

 

 

 

 

What is AIFRED's user journey?

  1. AIFRED contains two main parts, Data Preparation and Data Processing& Storage
  2.  
  3. Data Preparation:

       (1) The user opens the mobile application to check the status of the backend servers. 

       (2) The user scans the form from the device camera or uploads the image from the mobile gallery.

 The mobile application automatically performs a perspective transformation on all the forms that are scanned.

       (3) When necessary, the user adjusts the bounding boxes of the perspective transformation to ensure that the form image is captured correctly.

  1. Data Processing & Storage

      (1) Images will be uploaded to the AI server where our AI engine will perform form classification and information extraction, the extracted information will be sent back to the mobile app.

      (2) The AI engine returns alerts for fields that have low confidence predictions, the user will then be able to correct the data by consulting the original form.

      (3) After validating the data, the user then submits the extracted information with the necessary corrections to the storage server.

 

 

 

 

 

 

 

 

 

 

How well does AIFRED's AI perform?

 

(1) The Document classification model achieves 98.1% accuracy

 

(2) AIFRED only needs 5s for processing each document 

 

(2) AIFRED outperforms Google's AI model by achieving 88.9% of Key-Value Extraction Accuracy for Forms with fixed template(ARC) 

 

(2) AIFRED achieves 48.8% of Key-Value Extraction Accuracy for forms without fixed template(PO, AWB) which is comparable with Google's AI model 

 

 

 

 

 

 

 

 

 

 

 

 

Big thank you to:

photo 2021 07 27 21 26 58
screenshot 2021 07 31 at 11 09 33 pm

TEAM MEMBERS

student Alex Wang Wei Jie Information Systems Technology and Design
student Tan Jun Yong Engineering Systems and Design
student Shang Zewen Information Systems Technology and Design
student Jose Johnson Emerson Raja Information Systems Technology and Design
student Zhang Jiazheng Information Systems Technology and Design
student He Yuhang Information Systems Technology and Design
border border