Defense Notices

Fingerspelling in sign language has been the means of communicating technical terms and proper nouns when they do not have dedicated sign language gestures. The automatic recognition of fingerspelling can help resolve communication barriers when interacting with deaf people. The main challenges prevalent in automatic recognition tasks are the ambiguity in the gestures and strong articulation of the hands. The automatic recognition model should address high inter-class visual similarity and high intra-class variation in the gestures. Most of the existing research in fingerspelling recognition has focused on the dataset collected in a controlled environment. The recent collection of a large-scale annotated fingerspelling dataset in the wild, from social media and online platforms, captures the challenges in a real-world scenario. This study focuses on implementing a fine-grained visual attention approach using Transformer models to address the challenges existing in two fingerspelling recognition tasks: multiclass classification of static gestures and sequence-to-sequence prediction of continuous gestures. For a dataset with a single gesture in a controlled environment (multiclass classification), the Transformer decoder employs the textual description of gestures along with image features to achieve fine-grained attention. For the sequence-to-sequence prediction task in the wild dataset, fine-grained attention is attained by utilizing the change in motion of the video frames (optical flow) in sequential context-based attention along with a Transformer encoder model. The unsegmented continuous video dataset is jointly trained by balancing the Connectionist Temporal Classification (CTC) loss and maximum-entropy loss. The proposed methodologies outperform state-of-the-art performance in both datasets. In comparison to the previous work for static gestures in fingerspelling recognition, the proposed approach employs multimodal fine-grained visual categorization. The state-of-the-art model in sequence-to-sequence prediction employs an iterative zooming mechanism for fine-grained attention whereas the proposed method is able to capture better fine-grained attention in a single iteration.

With the rapid development of machine learning (ML) and deep learning (DL) methodologies, DL methods can be leveraged for wireless network reconfigurability and channel modeling. While deep learning-based methods have been applied in a few wireless network use cases, there is still much to be explored. In this project, we focus on the application of deep learning methods for two scenarios. In the first scenario, a user transmitter was moving randomly within a campus area, and at certain spots sending wireless signals that were received by multiple antennas. We construct an active deep learning architecture to predict user locations from received signals after dimensionality reduction, and analyze 4 traditional query strategies for active learning to improve the efficiency of utilizing labeled data. We propose a new location-based query strategy that considers both spatial density and model uncertainty when selecting samples to label. We show that the proposed query strategy outperforms all the existing strategies. In the second scenario, a reconfigurable intelligent surface (RIS) containing 4096 tunable cells reflects signals from a transmitter to users in an office for better performance. We use the training data of one user's received signals under different RIS configurations to learn the impact behavior of the RIS on the wireless channel. Based on the context and experience from the first scenario, we build a DL neural network that maps RIS configurations to received signal estimations. In the second phase, the loss function was customized towards our final evaluation formula to obtain the optimum configuration array for a user. We propose and build a customized DL pipeline that automatically learns the behavior of RIS on received signals, and generates the optimal RIS configuration array for each of the 50 test users.

Android's fragmented ecosystem makes the delivery of security updates and OS upgrades cumbersome and complex. While Google initiated various projects such as Android One, Project Treble, and Project Mainline to address this problem, and other involved entities (e.g., chipset vendors, manufacturers, carriers) continuously strive to improve their processes, it is still unclear how effective these efforts are on the delivery of updates to supported end-user devices. In this paper, we perform an extensive quantitative study (August 2015 to December 2019) to measure the Android security updates and OS upgrades rollout process. Our study leverages multiple data sources: the Android Open Source Project (AOSP), device manufacturers, and the top four U.S. carriers (AT\&T, Verizon, T-Mobile, and Sprint). Furthermore, we analyze an end-user dataset captured in 2019 (152M anonymized HTTP requests associated with 9.1M unique user identifiers) from a U.S.-based social network. Our findings include unique measurements that, due to the fragmented and inconsistent ecosystem, were previously challenging to perform. For example, manufacturers and carriers introduce a median latency of 24 days before rolling out security updates, with an additional median delay of 11 days before end devices update. We show that these values alter per carrier-manufacturer relationship, yet do not alter greatly based on a model's age. Our results also delve into the effectiveness of current Android projects. For instance, security updates for Treble devices are available on average 7 days faster than for non-Treble devices. While this constitutes an improvement, the security update delay for Treble devices still averages 19 days.

Solid-state transformers (SSTs) are comprised of several cascaded power stages with different voltage levels. This leads to more challenges for operation and maintenance of the SSTs not only under critical conditions, but also during normal operation. However, one of the most important reliability concerns for the SSTs is related to high voltage side switch and grid faults. High voltage stress on the switches, together with the fact that most modern SST topologies incorporate large number of power switches in the high voltage side, contribute to a higher probability of a switch fault occurrence. The power electronic switches in the high voltage stage are under very high voltage stress, significantly higher than other SST stages. Therefore, the probability of the switch failures becomes more substantial in this stage. In this research, a new technique is proposed to improve the overall reliability of the SSTs by enhancing the reliability of the high voltage stage.

The proposed method restores the normal operation of the SST from the point of view of the load even though the input stage voltages are unbalanced due to the switch faults. On the other hand, high voltage grid faults that result in unbalanced operating conditions in the SST can also lead to dire consequences in regards to safety and reliability. The proposed method can also revamp the faulty operation to the pre-fault conditions in the case of grid faults. The proposed method integrates the quasi-z-source inverter topology into the SST topology for rebalancing the transformer voltages. Therefore, this work develops a new SST topology in conjunction with a fault-tolerant operation strategy that can fully restore operation of the proposed SST in the case of the two fault scenarios. The proposed fault-tolerant operation strategy rebalances the line-to-line voltages after a fault occurrence by modifying the phase angles between the phase voltages generated by the high voltage stage of the proposed SST. The boosting property of the quasi-z-source inverter topology circuitry is then used to increase the amplitude of the rebalanced line-to-line voltages to their pre-fault values. A modified modulation technique is proposed for modifying the phase angles and controlling the quasi-z-source inverter topology shoot-through duty ratio.

Crowd estimation is an integral part of crowd analysis. It plays an important role in event management of huge gatherings like Hajj, sporting, and musical events or political rallies. Automated crowd count can lead to better and effective management of such events and prevent any unwanted incident. Crowd estimation is an active research problem due to different challenges pertaining to large perspective, huge variance in scale and image resolution, severe occlusions and dense crowd-like cluttered background regions. Current approaches cannot handle huge crowd diversity well and thus perform poorly in cases ranging from extreme low to high crowd-density, thus, leading to crowd underestimation or overestimation. Also, manual crowd counting subjects to very slow and inaccurate results due to the complex issues as mentioned above. To address the major issues and challenges in the crowd counting domain, we separately investigate two different types of input data: uni-modal (Image) and multi-modal (Image and Audio).

In the uni-modal setting, we propose and analyze four novel end-to-end crowd counting networks, ranging from multi-scale fusion-based models to uniscale one-pass and two-pass multi-task models. The multi-scale networks also employ the attention mechanism to enhance the model efficacy. On the other hand, the uni-scale models are equipped with novel and simple-yet-effective patch re-scaling module (PRM) that functions identical but lightweight in comparison to the multi-scale approaches. Experimental evaluation demonstrates that the proposed networks outperform the state-of-the-art methods in majority cases on four different benchmark datasets with up to 12.6% improvement in terms of the RMSE evaluation metric. Better cross-dataset performance also validates the better generalization ability of our schemes. For the multimodal input, the effective feature-extraction (FE) and strong information fusion between two modalities remain a big challenge. Thus, the aim in the multimodal environment is to investigate different fusion techniques with improved FE mechanism for better crowd estimation. The multi-scale uni-modal attention networks are also proven to be more effective in other deep leaning domains, as applied successfully on seven different scene-text recognition datasets with better performance.

Machine learning (ML) is transforming a wide range of applications, promising to bring immense economic and social benefits. However, it also raises substantial security and privacy challenges.  In this dissertation we describe a framework for efficient, collaborative and secure ML training using a federation of client devices that jointly train a ML model using their private datasets in a process called Federated Learning (FL). First, we present the design of a blockchain-enabled Privacy-preserving Federated Transfer Learning (PPFTL) framework for resource-constrained IoT applications. PPFTL addresses the privacy challenges of FL and improves efficiency and effectiveness through model personalization. The framework overcomes the computational limitation of on-device training and the communication cost of transmitting high-dimensional data or feature vectors to a server for training. Instead, the resource-constrained devices jointly learn a global model by sharing their local model updates. To prevent information leakage about the privately-held data from the shared model parameters, the individual client updates are homomorphically encrypted and aggregated in a privacy-preserving manner so that the server only learns the aggregated update to refine the global model. The blockchain provides provenance of the model updates during the training process, makes contribution-based incentive mechanisms deployable, and supports traceability, accountability and verification of the transactions so that malformed or malicious updates can be identified and traced to the offending source. The framework implements model personalization approaches (e.g. fine-tuning) to adapt the global model more closely to the individual client's data distribution.

In the second part of the dissertation, we turn our attention to the limitations of existing FL algorithms in the presence of adversarial clients who may carry out poisoning attacks against the FL model. We propose a privacy-preserving defense, named CONTRA, to mitigate data poisoning attacks and provide a guaranteed level of accuracy under attack.  The defense strategy identifies malicious participants based on the cosine similarity of their encrypted gradient contributions and removes them from FL training. We report the effectiveness of the proposed scheme for IID and non-IID data distributions. To protect data privacy, the clients' updates are combined using secure multi-party computation (MPC)-based aggregation so that the server only learns the aggregated model update without violating the privacy of users' contributions.

Swarms of unmanned aerial systems (UASs) usage is becoming more prevalent in the world. Many private companies and government agencies are actively developing analytical and technological solutions for multi-agent cooperative swarm of UASs.  However, majority of existing research focuses on developing guidance, navigation, and control (GNC) algorithms for swarm of UASs and proof of stability and robustness of those algorithms. In addition to profound challenges in control of swarm of UASs, a reliable and fast intercommunication between UASs is one of the vital conditions for success of any swarm.  Many modern UASs have high inertia and fly at high speeds which means if latency or throughput are too low in swarms, there is a higher risk for catastrophic failure due to intercollision within the swarm. This work presents solutions for scaling number of collaborative agents in swarm of UASs using frequency-based hierarchy. This work identifies shortcomings and discusses traditional swarm communication systems and how they rely on a single frequency that will handle distribution of information to all or some parts of a swarm. These systems typically use an ad-hoc network to transfer data locally, on the single frequency, between agents without the need of existing communication infrastructure. While this does allow agents the flexibility of movement without concern for disconnecting from the network and managing only neighboring communications, it doesn’t necessarily scale to larger swarms. In those large swarms, for example, information from the outer agents will be routed to the inner agents. This will cause inner agents, critical to the stability of a swarm, to spend more time routing information than transmitting their state information. This will lead to instability as the inner agents’ states are not known to the rest of the swarm. Even if an ad-hoc network is not used (e.g. an Everyone-to-Everyone network), the frequency itself has an upper limit to the amount of data that it can send reliably before bandwidth constraints or general  interference causes information to arrive too late or not at all.

We propose that by using two frequencies and creating a hierarchy where each layer is a separate frequency, we can group large swarms into manageable local swarms. The intra-swarm communication (inside the local swarm) will be handled on a separate frequency while the inter-swarm communication will have its own. A normal mesh network was tested in both hardware in the loop (HitL) scenarios and a collision avoidance flight test scenario. Those results were compared against dual-frequency HitL simulations. The dual-frequency simulations showed overall improvement in the latency and throughput comparatively to both the simulated and flight-tested mesh network.

In the last decade, the discovery of noncoding RNA (ncRNA) has exploded. Classifying these ncRNA is critical to determining their function. This thesis proposes a new method employing deep convolutional neural networks (CNNs) to classify ncRNA sequences. To this end, this thesis first proposes an efficient approach to convert the RNA sequences into images characterizing their base-pairing probability. As a result, classifying RNA sequences is converted to an image classification problem that can be efficiently solved by available CNN-based classification models. This thesis also considers the folding potential of the ncRNAs in addition to their primary sequence. Based on the proposed approach, a benchmark image classification dataset is generated from the RFAM database of ncRNA sequences. In addition, three classical CNN models and three Siamese network models have been implemented and compared to demonstrate the superior performance and efficiency of the proposed approach. Extensive experimental results show the great potential of using deep learning approaches for RNA classification.

In recent years, the problem of real-time scheduling has increasingly become more important as well as more complicated. The former is due to the proliferation of safety critical systems into our day-to-day life and the latter is caused by the escalating demand for high performance which is driving the multicore architecture towards consolidation of various kinds of heterogeneous computing resources into smaller and smaller SoCs. Motivated by these trends, this dissertation tackles the following fundamental question: how can we guarantee predictable real-time execution while preserving high utilization on heterogeneous multicore SoCs?

This dissertation presents new real-time scheduling techniques for predictable and efficient scheduling of mixed criticality workloads on heterogeneous SoCs. The contributions of this dissertation include the following: 1) a novel CPU-GPU scheduling framework, called BWLOCK++, that ensures predictable execution of critical GPU kernels on integrated CPU-GPU platforms 2) a novel gang scheduling framework called RT-Gang, which guarantees deterministic execution of parallel real-time tasks on the multicore CPU cluster of a heterogeneous SoC. 3) optimal and heuristic algorithms for gang formation that increase real-time schedulability under the RT-Gang framework and their extension to incorporate scheduling on accelerators in a heterogenous SoC. 4) A case-study evaluation using an open-source autonomous driving application that demonstrates the analytical and practical benefits of the proposed scheduling techniques.

So much of what we do on a daily basis is dependent on computers: email, social media, online gaming, banking, online shopping, virtual conference calls, and general web browsing to name a few. Most of the devices we depend on for these services are computers or servers that we do not own, nor do we have direct physical access to. We trust the underlying network to provide access to these devices remotely. But how do we know which computers/servers are safe to access, or verify that they are who they claim to be? How do we know that a distant server has not been hacked and compromised in some way?

Remote attestation is a method for establishing trust between remote systems. An "appraiser" can request information from a "target" system. The target responds with "evidence" consisting of run-time measurements, configuration information, and/or cryptographic information (i.e. hashes, keys, nonces, or other shared secrets). The appraiser can then evaluate the returned evidence to confirm the identity of the remote target, as well as determine some information about the operational state of the target, to decide whether or not the target is trustworthy.

A tool that may prove useful in remote attestation is the TPM, or "Trusted Platform Module". The TPM is a dedicated microcontroller that comes built-in to nearly all PC and laptop systems produced today. The TPM is used as a root of trust for storage and reporting, primarily through integrated cryptographic keys. This root of trust can then be used to assure the integrity of stored data or the state of the system itself. In this thesis, I will explore the various functions of the TPM and how they may be utilized in the development of the remote attestation language, "Copland".