BEYOND A SINGLE PIXEL Many home appliances and devices promise to make our homes safer and more comfortable, and to save us time or make our lives easier. They automate trivial tasks or make the controls of such devices available at our fingertips through connectivity to our mobile phone. This functionality is enabled by a plethora of digital sensors, ranging from devices that measure air temperature, humidity and light conditions to motion detectors, gas sensors, and more. From the perspective of the information that such sensors provide, they have one thing in common: They are point-wise and lack spatial resolution. For example, a conventional temperature or a light sensor would report the temperature or the light intensity only at the point of installation in a room. No awareness of the people in the room, their actions, or their level of comfort can be inferred from such sensors.
In contrast, visual imaging sensors have very fine spatial resolution. delivered by the millions of detectors integrated in the sensor. In that way, they generate rich, unstructured information, upon which visual image analytics are applied to obtain structured information (for example, not just “there are people at the doorbell” but how many and who). The application of visual image sensors in smart robots and home security system is already broad. Now, evolution of the technology is bringing other multipixel sensors to the arena of home automation and smart homes. Thermal imaging sensors are the prime candidates, with prospects for preventing fire hazards and guiding cooks in the kitchen, monitoring human occupancy without sacrificing privacy, measuring our comfort level, and remotely ensuring the well-being of infants and the elderly.
Below, we look at a few specific cases.
YOUR SAFER AND SMARTER KITCHEN Why thermal imaging in the kitchen? The short answer is for safety, health, and convenience. Indeed, according to the U.S. National Fire Protection Association (NPFA), cooking is the leading cause of domestic fires, accounting for 49% of such fires. Not all the affected households had a malfunctioning smoke detector; rather, by the time the smoke was detected, it was too late to eliminate the conditions leading to the fire. Thermal imaging can provide a clear path to minimizing such costly and devastating accidents by identifying elevated temperatures that may lead to fire long before the hazard conditions set in.
A natural way of providing thermal imaging in the kitchen is by integrating the sensor within the extractor hood to realize a cooktop monitor. Cooksy-Pro, shown in Figure 1, is the world’s first intelligent cooking assistant product. In the figure, Cooksy-Pro is the copper-colored device mounted under the extractor hood. From there, the thermal imager can continuously “see” the temperature distribution on the entire cooktop or stove.
Many culinary ingredients are sensitive to the temperature at which they are processed. Overheating them may not only deplete their nutritional value but even turn them into less than friendly compounds for our bodies. A thermal imager above the cooktop can “read” a cooking meal’s temperature continuously and indicate it to us, thereby helping us produce healthier meals.
“This leads us to the concept of a culinary assistant, helping you to create healthy and delicious meals every time,” said Steven Cartwright, CFO of Cooksy Corp. “In its essence, Cooksy enables you to share your culinary recipes or to follow other’s recipes to the minute details of how long and at what temperature the ingredients are being cooked — during the process itself, in real time. “
Cartwright explained that “Cooksy combines visual and thermal camera sensors with a sophisticated intelligence and connectivity in a device that is easy to mount above the cooktop. The device can livestream your cooktop to your mobile, letting you ‘see’ the temperature of the cooked ingredients, and advises you how and when to act upon this information.”
Figure 1: Cooksy-Pro is the world’s first intelligent cooking assistant. (Source: Cooksy Corp.)
The benefits of a cooktop monitor extend beyond culinary art and nutrition into the realm of kitchen safety. For example, the monitor can detect an unattended open flame of a gas cooker or an overheated range plate, overheated pots and pans with or without oil in them, boiling and overflowing containers, and other dangerous conditions. Figure 2 shows the boiling of a pot of water, with a lid, that is monitored over time by a thermal imager. Although the directly observable temperatures never approach 100 (212 ), the boiling point for water, the application of thermal image processing and analytics makes it possible to detect when the water starts to boil, when it starts to overflow from under the lid, and when the gas burner is turned off. In the context of smart homes and connected appliances, the identification of such events allows not only the triggering of an alarm but also the moderation or stopping of the electricity or gas supply if necessary to prevent an incident or fire hazard.
Figure 2: Thermal imaging analytic for cooktop applications (Source: Meridian Innovation)
A cooktop monitor is not the only place in the kitchen where one may find a thermal imaging sensor. An early prototype of a thermal-imaging–enabled microwave oven — originally presented by Meridian Innovation at CES 2020 — is shown in Figure 3 (a video of its operation can be seen at facebook.com/watch/?v=685705472262649).
Figure 3: Smart thermal imaging microwave oven (Source: Meridian Innovation)
The unit features the ability to heat the food to the desired temperature, instead of heating it for a fixed time at a fixed power. It is based on an off-the-shelf microwave oven that has been modified to integrate a thermal imager and alternative controls. This allows us to dispense with the guesswork we must frequently apply in selecting the power and the time for reheating our food. At long last, we can convey our intent directly, by selecting the desired food temperature, and leave the technicalities to the machine. This intent is conveyed either through a wirelessly connected mobile device or via the touchscreen integrated in the modified microwave oven. Naturally, one can set the desired food temperature as well as monitor the food while it is being heated up.
Moreover, with the advances of TinyML and other initiatives toward the availability of AI capabilities on embedded processors, this setup may allow for the unit to guess what we have put inside the microwave while it is being heated, so that the optimal temperature is automatically selected for us.
The imperative for a timely delivery of thermal imaging cameras for the detection of elevated temperature and the complexity of building a good solution have led to many partnership efforts to provide a number of reference designs for OEMs and ODMs. One good example is described in the article “Arrow Electronics Launches AI Thermal Sensing Solution Powered by STMicroelectronics’ X-CUBE AI” published at European Business Magazine https://europeanbusinessmagazine.com/media-outreach/arrow-electronics-launches-ai-thermal-sensing-solution-powered-by-stmicroelectronics-x-cube-ai/. Another new solution, achieved with our partner Generalplus Technology, is described below in the section titled “When thermal imaging meets AI chipset.”
AIR CONDITIONING THAT CARES FOR YOUR COMFORT HVAC systems are another example where integrating thermal imaging can make a positive impact. Most air conditioners in common use today embed a simple temperature sensor in the box to control the room temperature; this sensor provides the only feedback from the environment. Such systems can be regarded as Generation 1. In contrast, a second-generation system, integrating a thermal imager, knows whether there are occupants in the room and possibly where in the room those occupants are located. This enables more efficient utilization of the air-conditioning unit. For a small apartment room or a hotel room, a thermal imager of a few hundred to a thousand pixels may be sufficient, and for a larger living space, a sensor with a few thousand pixels is needed, coupled with a lens that has field of view of more than 90°.
Figure 4 shows the smart thermal imager air conditioner concept that Meridian Innovation showcased at the Shanghai Smart Home Technology Expo in September 2019. Note that the content of the thermal image gives insight into not only into the occupancy of the compartment, but also the overall distribution of temperature across the room. Based on this more detailed information, the unit can automatically adjust the temperature, direction, and speed of the airflow to save energy and improve the degree of comfort.
Figure 4: Smart thermal imager air conditioner (Source: Meridian Innovation)
As for comfort, there are various mathematical models that establish a direct relation between a person’s average skin temperature and the person’s perceived comfort level. We should keep in mind that the purpose of air-conditioning and HVAC systems is to ensure a certain level of comfort and well-being of the occupants, rather than to merely cool or heat the air of a room to a preset temperature. This leads us to the concept of what we call Generation 3 HVAC systems. At that tier, the heating or cooling unit uses even higher-resolution thermal imaging — on the order of 5,000 to 20,000 pixels — and can obtain a more detailed thermal picture, measure the average skin temperature of the occupants directly, and estimate their level of comfort. This information enables ambience control that aims at true thermal comfort, rather than temperature, and puts the human experience at the center of the feedback loop.
Generation 2 air-conditioning units are already available on the market, made possible by the recent volume availability of cost-efficient thermal imagers of 1,000 to 5,000 pixels. We project that the rapid evolution of thermal imaging technology will bring Generation 3 functionality to high-end models within five years.
WELLNESS MONITORING OF INFANTS AND ELDERS Baby monitors with video streaming have already been on the market for a while and are a great way to share moments of the child’s joy with relatives and friends, as well as to let the caregiver keep an eye on the baby remotely. However, the embedded visual cameras cannot give a direct insight into the physiological state of the baby. A thermal imaging sensor can.
There is a lot of research and development under way to create baby monitors that integrate both visual and thermal cameras and back the cameras with a rich AI computational engine. Figure 5 shows an example of the new products that are on the horizon. The field offers many possibilities.
For example, because elevated temperature is a common symptom of illness, it is every parent’s desire to recognize the symptom as early as possible, even in the middle of the night. To address this application, one can develop an accurate tracking of the baby’s normal temperature patterns when the baby is healthy and happy, then automatically detect the occurrence of anomalous temperature patterns and alert the child’s caregivers early in the development of a symptom. It should be kept in mind that the healthy pattern of babies’ core body temperature exhibits a significant diurnal variation that can be nearly 2° between its lowest and highest points. This pattern appears to be unique to each child and slowly evolves as the child’s physiology develops, so capturing the pattern requires continuous observation. Therefore, a baby monitor positioned above the crib seems like a natural place to observe and record this evolution.
Beyond health status, the thermal and visual light streaming modalities can be further leveraged in a feedback loop with a smart baby mobile to engage the developing brain in interactive play. Both the behavioral pattern and corresponding temperature markers can be analyzed at the same time. These technologies can help caregivers nurture safe, smart, and healthy babies.
Figure 5: Smart baby monitoring (Source: Courtesy of 5GenCare)
Wellness monitoring by thermal imaging can extends to the elderly too. This is crucially important given two significant global trends: increased life expectancy and aging of the population, and the increasing number of elderly people living alone. Thermal imaging may help in many ways, from simple monitoring of vital signs of temperature and motion all the way to complex behavioral analytics. One common and important theme in elder care is the detection and even the prevention of falls. While this can be achieved with the installation of visual cameras in the living room, one would not put cameras in a bathroom — and yet bathrooms pose the highest risk.
With this in mind, a team from LSCM Hong Kong pioneered a thermal camera solution, built on a relatively low-resolution LWIR sensor with just under 5,000 pixels. This resolution is high enough to allow for pose estimation but small enough to allow for continuous, real-time thermal image processing with modest computational power — and, most importantly, to preserve privacy. The unit locally runs a sophisticated algorithm able to detect the occurrence of an accident or a fall and alert a responsible nurse or family member about the incident. The product won a gold medal at the 47th International Exhibition of Inventions in Geneva in 2019.
Other companies are developing camera solutions that help nursing facilities prevent common accidents. When a resident gets out of bed, such units can detect that the person is awake and promptly alert a caretaker to attend.
WHEN THERMAL IMAGING MEETS AI CHIPSET To bring integrated reference designs to OEMs and ODMs, Meridian Innovation is collaborating with Generalplus Technology, a Taiwan-based global supplier of integrated circuits known for its consumer products and for home-appliance solutions used in millions of the industry’s top-brand products. “We are excited to partner with Meridian Innovation. Their thermal imaging module and Generalplus’s AI chip are a perfect solution for smart home Silicon Carbide Wide-bandgap semiconductors have the potential to revolutionize power electronics. This 145-page guide on Silicon Carbide (SiC), which brings together leading industry voices, examines the engineering challenges, market potential, and application opportunities. Hardcover - $ 150.00 Digital - $ 45.00 Add to cart PowerUP Expo Conference Proceedings This complimentary digital guide provides a comprehensive collection of articles and opinion pieces from keynotes, panel discussions, and technical presentations at the recent PowerUP virtual conference. $ 0.00 Add to cart Silicon 100: Startups Worth Watching in 2021 EE Times’ 21st revision of the Silicon 100, our annual list of electronics and semiconductor startups to watch, explores which sectors are attracting the most VCs and entrepreneurs and draws a comprehensive picture of semiconductor technology trends $ 25.00 Add to cart AV & EV Conference Proceedings This complimentary digital guide provides a comprehensive review of the virtual event, including recorded sessions and downloadable presentations. $ 0.00 Add to cart Gallium Nitride This 150+ page book on Gallium Nitride (GaN) power devices provides a comprehensive look at the technology, applications, market, and future of this emerging wide-bandgap material for power electronics. Hardcover - $ 150.00 Digital - $ 45.00 Add to cart Sensors in Automotive The 152-page anthology, with contributions from leading thinkers in the safety and automotive industries, heralds the industry’s progress and identifies the engineering community’s remaining challenges. Hardcover - $ 150.00 Digital - $ 45.00 Add to cart Silicon 100: Emerging Startups to Watch Every year (or so) we publish a list of electronics and semiconductor startups that grabbed our attention during the preceding year. This is our list for 2020, newly expanded from the Silicon 60 to the Silicon 100. $ 25.00 Add to cart 2020 Mind Of the Engineer The Mind of the Engineer study provides unrivalled visibility into the mindsets of today’s working engineers. The survey shows technology trends, challenges engineers face, resources they need and what they rely on most, and more. $ 15.00 Add to cart Oversized Mink Sherpa Blanket Cozy up with an AspenCore read, good cup of coffee and this mink Sherpa Blanket. Measuring 60”x70”, this micro mink/Sherpa blanket is a crowd pleaser. This luxurious blanket is perfect for home use, outdoor events, and much more. $ 45.00 Add to cart Stainless Steel EE Times Vacuum Coffee Cup 11 oz. stainless steel, double wall vacuum insulation technology in a stylish coffee mug with handle keeping your beverage of choice at the right temperature for hours. $ 20.00 Add to cart IoT & 5G World: Conference Proceedings This conference proceedings eBook is a collection of articles and opinion pieces from keynotes, panel discussions, and technical presentations at the recent IoT & 5G World virtual conference. $ 0.00 Add to cart . When Meridian’s engineer mounts its thermal imaging module on our AI SoC platform [Figure 6], it is a masterpiece in the making. A cost-effective and standalone solution is born to integrate temperature into audio, video, and graphics with convolutional neural network (CNN) functions for smart and AI applications. When presenting such a solution, we have seen great interest from customers in this cutting-edge technology to create an innovative product for better tomorrow,” said Jacky Chen, sales manager at Generalplus.
The 152-page anthology, with contributions from leading thinkers in the safety and automotive industries, heralds the industry’s progress and identifies the engineering community’s remaining challenges.
Figure 6: SmartForm Kit (Source: Courtesy of Generalplus)
CONCLUSION Thermal imaging sensors are poised to augment the functionality and value of many products in our homes, allowing us to solve long-standing challenges, such as wellness monitoring and fall and fire prevention, while raising the energy efficiency and comfort of our living spaces and even helping us make better meals. These advances are enabled by the volume availability of a new generation of cost-efficient thermal imagers with resolution of thousands and even tens of thousands of pixels.
At the same time, we see a proliferation of accessible reference hardware designs and software solutions that greatly facilitate adoption of the technology and enable fast development cycle of innovative products incorporating thermal imaging cameras. Indeed, this is pivotal. The co-founder of Meridian Innovation, a company that has been at the forefront of this revolution, expresses it this way: “Thermal imaging sensors are like the eyes of a person; they need a sophisticated intelligence behind [them], in order to make sense of all the thermal data that is coming through.”
Thus, the stage is set for the unfolding of yet another compelling story in which tech innovation fosters safer and better living
You must Register or Login to post a comment.
This site uses Akismet to reduce spam. Learn how your comment data is processed.