OUTPUTS

Phase 3 (January – May 2022)

WP3 – Transition preparation and continuation

Objectives:

(1) Final Gateway component prototype

The Gateway component is logically identified by a set of server programs and modules, each of which is responsible for a well-established task (bridge, server, database, user interface, client). At the hardware level, the Gateway is identified by a miniPC connected by cable or Wi-Fi to the home’s Ethernet network. Controllers for Z-Wave, ZigBee, Bluetooth communication subnets are also connected to this miniPC.

 

(2) Final Expert System component prototype

From an architectural point of view, the Expert System is located at the level of the Cloud Server and contains the following main components:

  1. Telemonitoring system module
    • Data aggregation sub-module
    • Data and project management sub-module
    • Monitoring scenarios and notifications configuration sub-module
  2. The virtual assistant
  3. Back-end reports

The Expert system communicates via the REST service component of the Cloud server with the other modules of the SMARTCARE application.

(3) Final DeveloperUI component prototype

The DeveloperUI module is implemented using Python as a back-end and HACS (Home Assistant Community Store) for the graphical interface to facilitate the discovery, installation, renewal, deletion of monitoring devices associated with SMARTCARE system beneficiaries. This module takes the form of a user interface and is designed to make it easier to design solutions for specific home care applications. The module was installed on Raspberry Pi 4 motherboards for the Info World pilot system, in a house and in a laboratory at the Gheorghe Asachi Technical University of Iași (Figure 13).

Figure 13 - SMARTCARE prototype sensors at Info World headquarters (left), in a house (center) and in a laboratory (right)

Figure 13 – SMARTCARE prototype sensors at Info World headquarters (left), in a house (center) and in a laboratory (right)

 

(4) Integration of hardware and software components and technical testing of the SMARTCARE system

The technical testing of the integration between the hardware and the bridge components involved testing the robustness of the bridge components in the event of failure of the connected hardware and had two stages:

  • data flow limit testing – dedicated software components have been created, along with drivers that have been switched to hardware emulation mode and erroneous data has been injected into bridge components. Validation was performed based on the configuration element generated / available at the start of the bridge containing the valid limits / sets of values for each data stream generated by the hardware;
  • behavioral testing in case of interruption of data flow from the hardware – the components of the hardware subsystems (including the main controllers for the Z-Wave and ZigBee networks) were disconnected one by one and interaction requests were generated from the gateway component. In the case of bridge components based on a web interface, values stored in the databases associated with the hardware systems were obtained, and in the case of the other hardware components, exceptions caused by a read / write time-out were treated.

(5) Defining evaluation scenarios and key performance indicators

Gateway component testing was performed using the data-oriented testing technique:

  • communication between each monitoring device and the gateway;
  • communication between the gateway and the expert system;
  • notification rules (according to WHO, EU standards);
  • case studies for hypertensive, diabetic, cardiac, COVID, obese, Alzheimer’s patients.

 

(6) Evaluation of the SMARTCARE platform through prototype applications

Three Philips Hue bulbs and three other Zipato 2 bulbs were used to compare with the smart bulb made in the previous project, i-Light (figure 14). The left side features a device that was created as part of the i-Light research project and includes sensors for measuring temperature, humidity and levels of volatile compounds. In the center is a commercially available smart light bulb, Philips Hue, while in the right is another commercially available smart light bulb, Zipato Bulb 2.

Figure 14 - Smart lighting device developed as part of the i-Light project (left), Philips Hue bulb (center) and Zipato 2 bulb (right)

Figure 14 – Smart lighting device developed as part of the i-Light project (left), Philips Hue bulb (center) and Zipato 2 bulb (right)

 

Raspberry Pi 4 Model B was used to collect the signal strength indicator (RSSI) values received from the three i-Light luminaires, three Philips Hue and three other Zipato bulbs, installed where they are marked with yellow circles in figure 15. A stable Wi-Fi connection was used between the luminaires and the system server. The monitored person had a Bluetooth enabled smartphone and was noticed by the Raspberry Pi devices of the installed system regarding the location. A RaZberry board has been attached to the Raspberry Pi devices so that it can communicate with Zipato bulbs using the Z-Wave Plus protocol. The system proved to be functional after being tested with all three types of luminaires.

Figure 15 - Apartment plan for tests that included i-Light luminaires, Philips Hue and Zipato bulbs to validate the technical system. The arrows show the user's movements (a) from the bed to the closet (blue); (b), from closet to bed (green); (c) from bed to desk (magenta)

Figure 15 – Apartment plan for tests that included i-Light luminaires, Philips Hue and Zipato bulbs to validate the technical system. The arrows show the user’s movements (a) from the bed to the closet (blue); (b), from closet to bed (green); (c) from bed to desk (magenta)

 

(7) Technical plan for project continuation

In the third stage of the SMARTCARE project, numerous experiments with hardware equipment were performed, as well as the testing of the software solution was involved.

First, a detailed analysis of the final components Gateway, Expert System and Developer UI was performed. Secondly, the integration of hardware and software components and the technical testing of the SMARTCARE system were performed. Evaluation scenarios and key performance indicators were defined, along with the evaluation of the SMARTCARE platform through prototype applications. The consortium has prepared the transition and continuation of the SMARTCARE project to be marketed.

Figure 16 – Synergy of the key elements for the technical plan for the continuation of the SMARTCARE project

Figure 16 – Synergy of the key elements for the technical plan for the continuation of the SMARTCARE project

Phase 2 (January – December 2021)

WP2 – Development, integration and evaluation of the final prototype

Objectives:
(1) Final Gateway component prototype

In stage 2 we studied from a hardware point of view the data about the patient and the environment in which he is. These are taken over by a heterogeneous network of sensors and transmitted to the Gateway for preprocessing, basic analysis and cloud storage (expert system). An express requirement in the requirements definition document was that the system be independent of communication media and protocols. Thus, the network contains sensors that communicate by wire or radio various communication protocols: ZigBee, Z-Wave, BLE, Wi-Fi. The gateway abstracts the physical size or event being monitored by disconnecting it from the device providing it. The proposed architecture allows at the runtime to add, remove, change a sensor in the network.

From a logical point of view, there are three sensor networks: a network of sensors for monitoring the vital parameters of the monitored person, a network of sensors for monitoring physical activity and a network of sensors / actuators for monitoring the environment (home automation). As stated, the separation is not performed at the physical level but at the logical level, at the consumer of information provided by the sensors, depending on its identifier.

Figure 5 – Hardware subsystem architecture for environmental automation – home automation


Figure 6 – Hardware subsystem architecture for monitoring vital parameters


Figure 7 – Subsystem hardware architecture for determining physical activity

(2) Final Expert System component prototype

The components of the Expert System are defined in such a way as to allow the visualization, configuration, management, aggregation and reporting of data and how end users will be able to view them in turn in the related graphical interfaces.

The Expert system, through a REST service, communicates with the other modules of the system (Gateway / DeveloperUI) and through its components, it aggregates, manages, controls and transmits the data to the database and graphical interfaces. Thus, with the help of the Expert System, the rules of logic and operation of the SMARTCARE application are managed.

Figure 8 – Overview of the interaction of the Expert System with the other SmartCare components

 

Figure 9 – Technologies used for the Expert System

(3) Final DeveloperUI component prototype

The interface of the DeveloperUI component is dedicated to users with the role of integrator or designer administrator, which offers the following functionalities: user management, configuration of user rights according to role; management of equipment and tools for configuring several aspects related to indoor location and monitoring and alert scenarios; project management; establishing business rules related to projects; viewing different types of reports.

Figure 10 – Testing the equipment in the graphical interface in case of motion detection, as well as the opening of the door

(4) Technical validation report of the final SMARTCARE prototype

The main purpose is to ensure that the hardware developed is fully interoperable with the software platform, that any hardware and software limit cases are covered at the functional level and that it does not cause widespread system errors.

Figure 11 – Local Gateway GUI – System Status, Manual Controls, Bridges / Devices / Connected Resources

(5) Test and evaluation plan

At this stage, we have planned to test the SMARTCARE system. Functional testing is primarily a black-box testing that checks the functionality of a system. The definition of test cases for functional testing is based on the specification. The functionality of a system is tested by stimulating it with various inputs, following the correctness of the system’s response to these inputs. Whenever needed, a gray box test will also be used. This type of testing is a combination of input-based black box testing and output verification and code-based white box testing. For the functional testing of SMARTCARE components, the following two main directions will be followed: breakdown of functionality, its isolation and individual testing (testing of functional units); general system testing (functional system testing).

(6) SMARTCARE prototype test and evaluation report

Partial testing and evaluation included the accuracy of indoor location when using commercially available smart light bulbs, namely Philips Hue, along with monitoring the evolution of heart rate based on a smart bracelet, FitBit Versa 2. We used commercially available smart bulbs to replace custom hardware previously developed in the i-Light project.

Figure 12 – Plan of the apartment used for the experiment. The arrows indicate the user’s movement (a) from the bed to the closet (blue); (b) from wardrobe to bed (green) and (c) from bed to desk (magenta); the user’s heart rate is correlated with movement, with samples taken displayed along the movement path

(7) Updated project website

Phase 1 (June – December 2020)

WP1 – SMARTCARE system design and development of the initial prototype

Objectives:
(1) System requirements

In stage 1, we studied the field of home autonomy support systems, identifying the system actors, the hardware required for the server and end users, and the use cases. In addition, the functional requirements for administrative activities, project management, patient monitoring by the system, use of the virtual assistant by patients and the reports generated by the system were determined. The activities for the notifications received by the owner and the doctor were established, together with the operations performed by the doctor. For each use case, determine the non-functional requirements, namely performance, interface, security and safety.

(2) Architectural design document

The architectural design of the SMARTCARE system is based on the description of the design objectives regarding the end user, the performance, the reliability and the development model. The objectives will be taken into account in the detailed implementation of the software architecture. An architectural scheme has also been defined, which will be the basis for the implementation of the hardware-software system. Server hardware, cloud services, and end users have been established, along with user management software, REST services, data aggregation, intelligent data analysis, custom monitoring and alert scenarios. Ways to generate monitoring reports, analyze environmental data and real-time alerts have been detailed. Responsibilities have been identified.

Figure 1 – Context diagram of the SMARTCARE system

(3) Initial Gateway component prototype

The Gateway component was analyzed from an architectural point of view, identifying bridges for interface modes, general structure and data, core interaction. These included the submodules for the Z-Wave and ModBus communication protocols, vital parameters, video and MQTT client. Other bridges, namely Core, West, East and North Bridge, will be responsible for efficient modularization of the application to provide high flexibility for further system expansion. Their roles are to facilitate the configuration, control of data and exceptions, data modeling and customization of the MQTT client. The knowledge base was detailed for video-monitored activities for patients with diabetes. A Z- Wave.Me UZB Smart Controller (shorturl.at/aqyFY) and a dimmer with a lighting channel (Everspring AD146 – shorturl.at/bnEQX) were used in the development and testing of the Z-Wave module. In the process of developing and testing the ModBus module, a USB-serial adapter was used to interface an RS485 serial bus to which two slave devices were connected (24-channel input module, 24-channel output module). In terms of image acquisition unit, recently tested video sensors that provide color and depth information have been tested. Following the tests performed, the candidate cameras selected for the SmartCare system are ZED 2 and ZED mini. Development at this time is not conditioned by the choice of one of the two image sensors. at / aqyFY) and a dimmer with a lighting channel (Everspring AD146 – shorturl.at/bnEQX). In the process of developing and testing the ModBus module, a USB-serial adapter was used to interface an RS485 serial bus to which two slave devices were connected (24-channel input module, 24-channel output module). As for the image acquisition unit, recently tested video sensors that provide color and depth information have been tested. Following the tests performed, the candidate cameras selected for the SmartCare system are ZED 2 and ZED mini. Development at this time is not conditioned by the choice of one of the two image sensors. at / aqyFY) and a dimmer with a lighting channel (Everspring AD146 – shorturl.at/bnEQX). In the process of developing and testing the ModBus module, a USB-serial adapter was used to interface an RS485 serial bus to which two slave devices were connected (24-channel input module, 24-channel output module). As for the image acquisition unit, recently tested video sensors that provide color and depth information have been tested. Following the tests performed, the candidate cameras selected for the SmartCare system are ZED 2 and ZED mini. Development at this time is not conditioned by the choice of one of the two image sensors. In the process of developing and testing the ModBus module, a USB-serial adapter was used to interface an RS485 serial bus to which two slave devices were connected (24-channel input module, 24-channel output module). As for the image acquisition unit, recently tested video sensors that provide color and depth information have been tested. Following the tests performed, the candidate cameras selected for the SmartCare system are ZED 2 and ZED mini. Development at this time is not conditioned by the choice of one of the two image sensors. In the process of developing and testing the ModBus module, a USB-serial adapter was used to interface an RS485 serial bus to which two slave devices were connected (24-channel input module, 24-channel output module). As for the image acquisition unit, recently tested video sensors that provide color and depth information have been tested. Following the tests performed, the candidate cameras selected for the SmartCare system are ZED 2 and ZED mini. Development at this time is not conditioned by the choice of one of the two image sensors. Recently tested video sensors that provide color and depth information have been tested. Following the tests performed, the candidate cameras selected for the SmartCare system are ZED 2 and ZED mini. Development at this time is not conditioned by the choice of one of the two image sensors. Recently tested video sensors that provide color and depth information have been tested. Following the tests performed, the candidate cameras selected for the SmartCare system are ZED 2 and ZED mini. Development at this time is not conditioned by the choice of one of the two image sensors.

The detection and classification of the activities of the monitored patient is performed in two stages:

  1. In the first instance, the person and objects of interest involved in these activities (floor, stairs, pizza, glass) are detected. Object detection can be done using models designed for both real-time applications and more complex models with slower processing speeds. Tested: YOLO v4 – 33 fps and Faster RCNN – 2 fps on an Nvidia Titan RTX architecture.
  2. Using the previous result, but also other characteristics such as spatial configuration, triplets of subject-predicate-object form are produced using a model for the detection of human-object interaction. In this regard, the iCAN architecture has been tested and tests using TNI and HAKE are being considered. The result can be interpreted in order to identify the monitored activities: serving food – the person is sitting at the table, the person is holding a fork, the person is eating pizza.

Figure 2 – Gateway architecture

(4) Initial Expert System component prototype

In stage 1, the functionalities of the Expert System component, the architecture, the data aggregation sub-modules, the intelligent data analysis, the monitoring and alert scenarios were established. It was presented the generation of reports for the activity of users, their location, biophysical parameters and medical devices, environmental conditions and alerts. The testing of the Expert System component was done by developing an application using Node.js, Express.js, Angular.js and Apache CouchDB technologies.

Figure 3 – Communication between Expert System / Gateway / Devices

(5) Initial DeveloperUI component prototype

During this stage, the functionalities of the DeveloperUI module on managing users, equipment and tools, indoor location, monitoring and alert scenarios, establishing business rules related to projects, along with viewing reports were identified, implemented and tested. The integrations made to work the tested system involved the use of software deConz, HACS, web service Meteorologisk Institutt to retrieve weather data, Mobile App to access the application on the mobile phone, Raspberry Pi Power Supply Checker to allow verification of the source of Raspberry Pi power supply that sends data from the test panel to the DeveloperUI module and the Z-Wave protocol. The interior location was studied in two real situations, using smart bulbs developed by the Spanish partner in the i-Light project. The SMARTCARE system is based on the adaptation of the i-Light solution, so that an article was published in this direction.

Figure 4 – The main user interface of the administrator type

(6) Project website

PRODUCTS

Final prototype of the Gateway component

Gateway is logically identified by a set of server programs and modules, each responsible for a well-established task (bridge, server, database, user interface, client).

 

Final prototype of the Expert System component

Expert System components allow you to view, configure, manage, aggregate, and report data and how end users can view it in their graphical interfaces.

 

Final prototype of the DeveloperUI component

DeveloperUI component as a graphical user interface and is intended to facilitate the design of solutions for specific home care applications.

 

Test and evaluation plan

SMARTCARE system testing planning is a combination of input-based black box testing, output verification, and code-based white box testing.