← Back

Technologies used on the project

We strive to use cutting-edge technologies for autonomous robot movement and fire detection.

Tehnologies

What we use in our project

Latest FireBot iterration

Autonomous driving

The FireBot is equipped with modern technologies and state-of-the-art navigational methods that enable autonomous navigation, route planning and obstacle avoidance. We are using Real-Time Appearance-Based Mapping (RTAB-Map), a state-of-the-art SLAM algorithm paired with LiDAR, and a depth camera which is used for mapping and navigation.


Machine Learning

The FireBot utilizes an RGB camera together with a modern convolutional neural network (CNN) for fault and fire detection. Firebot also utilizes various sensors for analyzing temperature, air humidity, and the presence of gasses released during combustion. We use semantic segmentation with three classes to detect flames and smoke on the RGB images, and to mark a normal enviroment. For ML workload and dataset pre-processing we are using NVIDIA Data Loading Library (DALI) together with NVIDIA RTX GPUs.


Thermal imaging

The FireBot also utilizes infrared thermal (IRT) camera that detects the heating of crucial equipment inside the office or warehouse at predefined patrol points. We are using basic image processing inside OpenCV that compares features from previous thermal images that the robot took on his patrol path.

Articles

RESEARCH, INNOVATION, DEVELOPMENT

The firebot team works in an academic institution, so our research findings in artificial intelligence, thermal imaging, and autonomous driving are exposed to the academic community in scientific articles.

  • FireBot - An Autonomous Surveillance Robot for Fire Prevention, Early Detection and Extinguishing

    Every year, fire is responsible for numerous deaths, as well as huge material losses. Therefore, prevention and early detection of fire have become a priority for society, as well as the main research and development issue for many scientists and various industries. This paper describes our work in the development of FireBot, an autonomous surveillance robot. The Firebot is equipped with modern technologies and state-of-the-art navigational and computer vision methods that enable autonomous navigation, obstacle avoidance, video surveillance, fire prevention and detection, and fire extinguishing. It utilizes both infrared thermal (IRT) and RGB cameras paired with a modern convolutional neural network (CNN) for fault and fire detection, as well as various other sensors for analyzing air composition, processing of surrounding sounds, and detecting irregularities in its environment in general. The best performing CNN was implemented and tested in real-world environments for fire detection purposes, the results of which are presented in this paper. A state-of-the-art SLAM algorithm paired with LiDAR and a depth camera is used for mapping and navigation. The architecture presented in this paper, along with all functionalities planned for future work, represents an innovative autonomous surveillance system that will make a great contribution in the field of fire prevention and detection.

    Authors: Balen, Josip; Damjanović, Davor; Marić, Petar; Vdovjak, Krešimir; Arlović, Matej

  • Modern CNNs Comparison for Fire Detection in RGB Images

    Every year, fire causes thousands of deaths as well as billions of dollars of material damage. Prevention and early fire detection have become a topic of interest for many scientists. While there are many existing solutions such as smoke detectors, flame detectors, chemical sen- sors, infrared thermal cameras and many other hybrid systems, computer vision techniques that use raw RGB image as an input have emerged as fast, reliable, precise, and economical enough to be widely used with a satisfactory accuracy. For that purpose, Convolutional Neural Networks (CNNs) were considered as they can take input image from an RGB camera, learn its features and classify it as fire or non-fire. Another im- portant thing to consider is their ability to be used on hardware with a limited amount of computational power, e.g. embedded systems. In this paper, four different versions of MobileNet, four versions of ResNet, and four versions of EfficientNet were evaluated by comparing their ability to detect fire while also taking into consideration their need for com- putational power. The evaluation was performed on a custom dataset that contains over 60,000 images. Overall, ResNet showed the lowest performance which was somewhat expected as it is the oldest network. MobileNets and EfficientNets showed similar performance proving them- selves to be capable when used as a fire detection classifiers. Also, due to their low number of parameters and low computational need, they are suitable for use in systems with limited resources.

    Authors: Vdovjak, Krešimir; Marić, Petar; Balen, Josip; Grbić, Ratko; Damjanović, Davor; Arlović, Matej

  • Obtaining Infrared Thermal Camera Sensor Calibration Data for Implementation in FireBot Autonomous Fire Protection Robot System

    Fire protection is one of the activities that follow the development of technology in realtime and implements all the innovations of a detection system. This paper presents a unique solution for the development of an autonomous robot for the prevention, detection, and extinguishing of fires by studying the problem of choosing the optimal early-detection sensor in the infrared part of the spectrum, which characterizes the highest level of excitation in the state of prevention. The robot is equipped with several different sensors arranged in a hierarchical structure. Thermal detection has proven to be a significant investment that can be adapted to the different complexity of the objects to be protected, taking into account image processing and modular implementation of the required sensors. To this end, it is necessary to calibrate systems for different thermal cameras. The calibration procedure on seven cameras and two pyrometers resulted in data required for inputdata correction and anomaly detection. The results of the analysis confirmed that devices of a higher price range have a lower deviation from the reference value compared to low-cost technical solutions. At the same time, results were observed indicating malfunction of more expensive devices, whose data exceed the specified nominal accuracy. Thanks to the performed calibration procedure and the obtained results, the observed problem is not an obstacle for implementation in an autonomous robotic system and can be used to correct the input data required for computer analysis.

    Full Paper

    Authors: Balen, Josip; Glavaš, Hrvoje; Vdovjak, Krešimir; Jakab, Josip

  • Radiometric Data Estimation using Thermogram and Comparison to the Data Provided by the Camera

    Fire causes many casualties as well as material damage and thus prevention and early fire detection are important. Many existing solutions can detect anomalies after other visible symptoms which is already too late. By using infrared thermal (IRT) cameras anomalies can be detected in the early stage and thus potential catastrophe can be prevented. For successful development and implementation of the anomaly detection algorithm, detailed spatial radiometric data is necessary. Most IRT camera manufacturers provide full radiometric data through their very limited software. In this paper, we estimate radiometric data using only the pixel intensity of the thermogram and temperature boundaries. Estimated data is further compared to the data provided by the camera manufacturer to validate our findings.

    Full Paper

    Authors: Vdovjak, Krešimir; Marić, Petar; Balen, Josip; Glavaš, Hrvoje

  • A Large Scale Dataset For Fire Detection and Segmentation in Indoor Spaces

    Fire represents a dangerous event, especially in inhabited areas where it can cause extensive economical damage, as well as take human lives, and therefore early fire detection is of utmost importance and requires careful attention. Utilizing images from security cameras and computer vision algorithms, it is possible to detect and raise the alarm in the event of a fire. The existence of similar-colored lights to the flame’s color is the great- est obstacle to indoor fire detection when it comes to computer vision. The lights may trigger false positive detections, resulting in false alarms and potential fire suppression by automated systems. By developing a new fire dataset for the training of deep neural networks, we attempted to circumvent the stated issue. Our dataset includes images of different colored lights, images with reflections of light that resemble the color of fire, and images of fire in a variety of environments, including warehouses, factories, shopping malls, residential buildings, offices, etc. Although there are numerous scientific papers and datasets for fire detection, there are not many datasets containing images of indoor fires. In this paper, we show a process of collecting and annotating images representing indoor fire in a manner suitable for deep neural network training. Furthermore, we present the developed Fire Sense image annotation tool and the process of image annotation. The dataset currently consists of more than 11000 annotated images of various types of fires in different environments.

    Full Paper

    Authors: Marić, Petar; Arlović, Matej; Balen, Josip; Vdovjak, Krešimir; Damjanović, Davor