Continual Learning for Robust Video Segmentation of Robot-Assisted Surgical Tool

WanDB Report

A Study on Regularization-Based Continual Learning Methods for Indic ASR

WanDB Report

Medical Multi-Modal Fusion & Cross Modal Alignment

Summer internship report

Dataset Analysis and Performance Report

This project focuses on analyzing and improving the performance of models using a multimodal dataset comprising Electronic Health Records (EHR) data, including demographic information, ICU vitals, Chest X-rays (CXR), ECG signals, and clinical notes. The primary tasks include predicting readmission, mortality, and length of stay (LOS) for patients based on data available within 24 to 48 hours after admission.

Modalities and Tasks

• EHR Demographics:

  Includes categorical data like race and age, with age discretized into groups due to deidentification.

• ICU Vitals:

  A selection of 39 vital signs, time series data, and categorical events like procedures.

• CXR:

  Time series of chest X-ray images.

• ECG:

  12-lead ECG signals represented as time series.

• Clinical Notes:

  Series of notes, such as discharge and radiology notes.

The tasks involve:

1. Readmission Prediction:

   Whether a patient will be readmitted within a month after discharge.

2. Mortality Prediction:

   Predicting if the patient will survive or die.

3. Length of Stay Prediction:

   Whether the stay will exceed 3 or 7 days.

Dataset and Class Imbalance

• The dataset shows heavy class imbalance across all tasks and modalities. This imbalance persists even when considering the intersection of multiple modalities, leading to a significant portion of the data being unused.

• The dataset was split into training, validation, and test sets with an approximately 80%-10%-10% split, maintaining the proportions of each modality combination and balancing the classes.

Experimentation Setup

• Data Pipeline:

  A heavily modified HADM pipeline handles data loading and interlinked modality fusion. PyTorch Lightning with Distributed Data Parallel (DDP) is used for training, with configurations managed via YAML files.

• Metrics:

  Evaluation metrics include binary classification metrics monitored across train, validation, and test sets. Performance is also assessed at the modality and modality combination levels.

• Model Structure:

  The model pipeline includes a modality encoder, a time series encoder, and a binary classification head. Class imbalance is addressed through oversampling and undersampling strategies, depending on the skewness of the class distribution.

Observations from Literature

• Cross-modal alignment is rarely performed or analyzed extensively in current research, and ECG data is often excluded or underutilized in multimodal tasks. Reported F1 scores are typically below 0.7, highlighting the challenge in improving model performance.

Experimentation Results

4.2. Multi-Modal Learning Framework

1. EHR:

a. GRU-D vs RNN:

Table 1: GRU-D vs RNN F1-score comparison.

MODEL	TEST F1 SCORE
GRU-D	0.6625
RNN	0.71844

RNN demonstrates superior performance compared to GRU-D, both in terms of F1 score and training time. RNN achieved a higher F1 score (0.71844 vs. 0.6625) and was more efficient, taking almost 40% less time to train. This indicates that the RNN model is both faster and more effective for EHR data

b. RNN vs RNN-TS:

Table 2: RNN vs RNN-TS F1-score comparison.

MODEL	TEST F1 SCORE
RNN	0.71844
RNN-TS	0.71549

RNN-TS (with time series data) performed almost similarly to the standard RNN (F1 score of 0.71549 vs. 0.71844). This suggests that incorporating time series data as positional embeddings does not significantly enhance the model's performance and may even slightly degrade it.

c. RNN Class Balance vs. Without Class Balance:

Table 3: RNN Class Balance comparison.

MODEL	TEST F1 SCORE
RNN (NO CLASS BALANCE)	0.71288
RNN (WITH CLASS BALANCE)	0.71844

Class balancing provided a marginal improvement in performance, boosting the F1 score from 0.71288 to 0.71844. This suggests that while class balancing helps, its impact is not substantial for the EHR dataset.

d. Training Time:

Table 4: GRU-D vs RNN-TS training time.

MODEL	TRAINING TIME (SECONDS)
GRU-D	135,230
RNN-TS	82,968

RNN-TS is significantly faster to train compared to GRU-D, indicating that the simpler RNN architecture is more efficient in this context, potentially due to the reduced complexity in handling time-series data compared to GRU-D.

2. CXR:

Table 5: Class balance and positional embedding comparison.

CONFIGURATION	TEST F1 SCORE
No class balance and positional embedding	0.56609
With class balance and positional embedding	0.66634
With class balance but no positional embedding	0.62633

Class balancing improved the model’s performance significantly, increasing the F1 score from 0.56609 to 0.66634. Additionally, adding positional embeddings further enhanced the performance (F1 score: 0.66634 vs. 0.62633), suggesting that positional embeddings in CXR data play a role in improving the model's capacity to capture temporal information.

3. Notes:

Table 6: Frozen vs unfrozen configurations.

CONFIGURATION	TEST F1 SCORE
No class balance (Frozen)	0.12199
Frozen (With class balance)	0.5004
Frozen (No positional embedding)	0.24544
Unfrozen (No class balance)	0.66085
Unfrozen (With class balance)	0.63192

Unfreezing the notes encoder and allowing the model to fine-tune the notes embeddings significantly improved performance, with the F1 score rising to 0.66085 (without class balance). Class balancing marginally reduced performance in the unfrozen model (0.63192). Frozen models performed poorly, highlighting the importance of fine-tuning. Additionally, positional embeddings play a significant role, as removing them drastically dropped the F1 score from 0.5004 to 0.24544.

4. ECG:

Table 7: Class balance impact on ECG performance.

CONFIGURATION	TEST F1 SCORE	AUROC
Without class balance	0.0	0.5197
With class balance	0.6236	0.52089

Without class balancing, the ECG model failed to predict more than one class, resulting in an F1 score of 0. However, class balancing greatly improved the performance, yielding an F1 score of 0.6236. This demonstrates that class imbalance severely impacts the ECG modality, and balancing is crucial for achieving meaningful predictions. Additionally, this highlights that AUROC is not a reliable metric in skewed predictions, as the AUROC for the unbalanced ECG model was still comparable despite the F1 score being 0.

5. Total Comparison:

Table 8: Summary across modalities.

MODALITY	TEST F1 SCORE	AUROC SCORE
Notes	0.66085	0.62601
ECG	0.6236	0.52089
RNN-TS (EHR)	0.71549	0.77536
CXR	0.66634	0.52081

Among the single modalities, the EHR data using RNN-TS performed the best, with an F1 score of 0.71549 and AUROC of 0.77536. CXR also performed well, with an F1 score of 0.66634. Notes and ECG followed with slightly lower scores. The AUROC scores indicate that while F1 remains the primary measure of interest, AUROC is not as reliable in imbalanced datasets like ECG and CXR.

6. Multimodality:

Table 9: Fusion method comparison.

FUSION METHOD	F1 SCORE
Sum Fusion of CXR + EHR	0.69789
Transformer Fusion of CXR + EHR	0.70182
Sum Fusion of CXR + EHR + Notes	0.45058
Transformer Fusion of CXR + EHR + Notes	0.66817
Sum Fusion of CXR + ECG + EHR + Notes	0.51967
Transformer Fusion of CXR + ECG + EHR + Notes	0.61993

The transformer-based fusion consistently outperforms the sum fusion method across all modality combinations. For instance, Transformer CXR + EHR (F1 score: 0.70182) slightly outperforms Sum CXR + EHR (F1 score: 0.69789). Similarly, Transformer CXR + ECG + EHR + Notes (F1 score: 0.61993) shows significant improvement over the sum fusion of the same modalities (F1 score: 0.51967). This indicates that transformer-based fusion is more effective in learning from the complex interactions between modalities.

Table 10: Fusion vs. single modality baselines.

FUSION/MODALITY	F1 SCORE
Transformer CXR + ECG + EHR + Notes	0.61993
Transformer CXR + Notes + ECG	0.55538
Transformer CXR + EHR + ECG	0.45439
Transformer CXR + EHR + Notes	0.66817
Transformer CXR + Notes	0.64509
Transformer Notes + ECG	0.58547
SINGLE MODALITY BASELINE
EHR	0.71549
CXR	0.66634
Notes	0.66085
ECG	0.62360

Although multimodal fusion using transformers improves performance in some cases, adding more modalities does not always result in better outcomes. For example, Transformer CXR + ECG + EHR + Notes (F1 score: 0.61993) performed worse than EHR alone (F1 score: 0.71549). This suggests that poorly aligned modality combinations can degrade performance. The best-performing multimodal models should ideally surpass the highest-performing single-modality model. However, this is not consistently observed, indicating that proper modality alignment and fusion are critical for achieving optimal performance. Transformer-based fusion provides better results compared to sum fusion in multimodal learning, especially when integrating complex and diverse modalities such as CXR, ECG, EHR, and Notes. However, simply combining modalities does not guarantee improved performance; well-aligned models must be carefully designed to achieve results that exceed the best-performing individual modalities. Ideally, the performance of a multimodal model should be greater than the maximum F1 score of the individual modalities (e.g., EHR or CXR alone).

WanDB Report

Close

Surgical Motion Planning

Project Summary:

The project focuses on the development of an embodied robotic agent designed to execute high-level natural language surgical instructions. This system leverages a combination of Reinforcement Learning (RL), Imitation Learning, and a pre-trained Large Language Model (LLM) to function as a "planner." The agent can perform short-horizon skills across multiple surgical scenarios using a library of pre-trained policies. These policies are selected based on the scenario, with skills that are common across scenarios mapped under common primitives with textual labels.

Methodology:

1. Planning and Filtering: The LLM is responsible for planning by determining the sequence of actions required to execute a given surgical instruction. A filtering process similar to SayCAN is used, which ranks skills based on both logical reasoning and geometric feasibility. The LLM generates possible action permutations, which are then filtered to a manageable number using a Chain-of-Thought (CoT) process, reducing the likelihood of errors and improving interpretability.

2. Action Scoring: Each action is scored based on two criteria: the LLM's planning score and the RL policy’s affordance score, which reflects the geometric feasibility of the action. The best action is selected based on these combined scores, with a fallback to the highest LLM score if all actions are deemed infeasible.

3. Execution: Once an action is selected, a human-in-the-loop is given the option to approve or modify the action before execution. The agent updates the scene and action status after each step, which informs subsequent actions.

4. Experimental Setups: The system is tested in two main environments: Surrol and Lapgym, each with specific tasks like retracting tissue, reaching a tumor, and picking and placing gauze. The experiments explore various strategies, including closed-loop, open-loop, and inner monologue setups, with scene and action updates. The DINO model is utilized for detecting objects in the surgical environment, such as gauze and blood, by generating bounding boxes around these targets based on textual inputs provided by the LLM. This integration allows for accurate, real-time detection of key elements, which is critical for the success of surgical tasks.

Key Features:

Human-in-the-loop Feedback: Ensures control over the actions performed, allowing for adjustments based on the requirements.

Chain-of-Thought (CoT): Improves planning efficiency and makes the process more interpretable for surgical tasks.

Scene Description Generation: Simplifies the scene context for the LLM, making the planning process more effective and relatable to real-world applications.

The project demonstrates the feasibility of integrating LLMs with RL, Imitation Learning, and DINO for complex surgical tasks, providing a robust foundation for further development in autonomous surgical systems.

Demo

Close

SaSi: A Self-augmented and Self-interpreted for Few-shot Cryo-ET Particle Detection

Description:

This project tackles the critical task of particle localization and classification in cryo-electron tomography (cryo-ET), focusing on real-world challenges such as limited and weakly annotated data. The study is conducted on both private datasets and the SHREC 2021 simulated dataset, aiming to establish robust baselines and explore advanced techniques to improve model performance. The goal is to enhance the accuracy and robustness of particle detection and classification, making it more effective for structural biology applications.

Methodology:

1. Supervised Learning: The approach begins by generating pseudo-strong labels from weak labels, converting them into segmentation masks by drawing spheres of minimum radius around the particle centroids. These masks serve as ground truth for training. The tomograms are divided into smaller subtomograms, which are fed into the model to generate voxel-level probabilities for each particle type. Key techniques include:

• Balanced Sampling and Window Method: Ensuring that training data is evenly sampled and that particle centroids are accurately localized within smaller windows.
• AugMix Modification: A tailored version of AugMix is applied to improve data augmentation, focusing on enhancing the diversity and robustness of the training samples.

2. Self-Supervised Learning: To further improve the model's performance, SimCLR is employed. This technique leverages contrastive learning to learn representations that are invariant to data augmentation, improving the model’s ability to generalize from few-shot data.

3. Post-Processing: After initial predictions, advanced post-processing techniques are applied to refine the results:

• Connected Components Analysis: Used to identify and isolate individual particles from the segmentation masks, improving the accuracy of particle localization.
• MeanShift Clustering: Applied to group detected particles based on their spatial coordinates, helping to better classify and localize particles within the tomogram.

WanDB Report

Close

SAM-PM: Enhancing Video Camouflaged Object Detection

• PDF
• Code
• bibtex

Abstract:

In the domain of large foundation models the Segment Anything Model (SAM) has gained notable recognition for its exceptional performance in image segmentation. However tackling the video camouflage object detection (VCOD) task presents a unique challenge. Camouflaged objects typically blend into the background making them difficult to distinguish in still images. Additionally ensuring temporal consistency in this context is a challenging problem. As a result SAM encounters limitations and falls short when applied to the VCOD task. To overcome these challenges we propose a new method called the SAM Propagation Module (SAM-PM). Our propagation module enforces temporal consistency within SAM by employing spatio-temporal cross-attention mechanisms. Moreover we exclusively train the propagation module while keeping the SAM network weights frozen allowing us to integrate task-specific insights with the vast knowledge accumulated by the large model. Our method effectively incorporates temporal consistency and domain-specific expertise into the segmentation network with an addition of less than 1% of SAM's parameters. Extensive experimentation reveals a substantial performance improvement in the VCOD benchmark when compared to the most recent state-of-the-art techniques. Code and pre-trained weights are open-sourced at https://github.com/SpiderNitt/SAM-PM

Methodology
The proposed SAM-PM framework adapts the Segment Anything Model (SAM) for the Video Camouflaged Object Detection (VCOD) task. SAM-PM consists of two main components: the Temporal Fusion Mask Module (TFMM) and the Memory Prior Affinity Module (MPAM).

1. Temporal Fusion Mask Module (TFMM): This module enhances temporal information by integrating mask embeddings from multiple frames. It uses a spatio-temporal cross-attention mechanism to create temporally enriched mask embeddings.
2. Memory Prior Affinity Module (MPAM): MPAM utilizes the temporally infused mask embeddings from TFMM along with image embeddings from the current and previous frames. It applies affinity to strengthen temporal consistency in mask predictions.
SAM-PM operates in a semi-supervised manner, where only the first frame's ground truth mask is used for training. The framework keeps SAM's weights frozen and trains only the SAM-PM components, ensuring parameter efficiency with less than 1 million parameters.

During training and inference, SAM-PM updates its memory with new frames and their predicted masks while discarding outdated data to maintain temporal coherence.

WanDB Report

Close Github

Test-time adaptation for Optical Flow

Optical Flow Estimation

In this research, we focused on optimizing the existing RAFT (Recurrent All-Pairs Field Transforms) model for optical flow estimation using a multi-GPU and multi-node setup. The model was deployed in a distributed computing environment using Distributed Data Parallel (DDP) processing, facilitated by Lightning-Fabric. The primary aim of this setup was to efficiently scale the model's training to accommodate larger batch sizes without running into out-of-memory (OOM) errors, a common challenge in deep learning tasks with high computational demands like optical flow estimation. The RAFT model's architecture was not modified; instead, we concentrated on improving the scalability and memory handling to enhance computational efficiency.

To achieve these optimizations, a cluster of multiple GPUs and nodes was configured to maximize parallel processing capabilities. The use of DDP allowed us to distribute the workload across multiple GPUs and nodes, synchronizing their operations to ensure smooth and efficient model training. This setup provided the necessary computational resources to handle large-scale data while maintaining memory efficiency across devices.

Several key optimization techniques were employed to maximize the efficiency and scalability of the RAFT model in this distributed environment. First, Fairscale’s CPU offloading was integrated to shift some memory loads from the GPU to the CPU, thereby freeing up GPU resources for more critical computations. This technique helped mitigate memory bottlenecks, allowing for the handling of larger batch sizes.

Next, we applied mixed-precision training, which combines 16-bit and 32-bit floating point operations. This significantly reduced memory consumption without compromising the accuracy of the optical flow estimations.

Finally, we implemented activation checkpointing to further minimize memory usage. By only storing key intermediate results and recomputing others when necessary, we reduced the overall memory footprint during training, allowing for a higher batch size without OOM errors.

These optimizations, applied in tandem with a distributed multi-GPU, multi-node setup, enabled us to expand the computational capabilities of the RAFT model, ensuring more efficient training while preserving memory integrity.

We begin with an input image \( x \), from which an initial optical flow map is generated:

\[ \text{opt} = \text{model}(x) \tag{9.1} \] Following this, the optical flow map is iteratively refined through a series of augmentation steps. Each step involves generating an augmented input \( x' \) as follows:

\[ x' = \text{aug}(\text{opt}, x) \tag{9.2} \] The model then processes the augmented input \( x' \) to produce an updated optical flow map:

\[ \text{opt}' = \text{model}(x') \tag{9.3} \] The loss function is calculated as the variance between the original optical flow map \( \text{opt} \) and the augmented optical flow map \( \text{opt}' \):

\[ \text{Loss} = \text{Variance}(\text{opt}', \text{opt}) \tag{9.4} \] The primary goal of this algorithm is to reduce the variance between the optical flow map generated for the original input and the augmented images. The augmentation process generates random patches with pixel values constrained by the minimum and maximum values of the input image. These patches are then placed in random positions in both the original image \( x \) and its augmented counterpart \( x' \). The optical flow map produced by the model during test-time adaptation guides the patch placement, and minimizing the variance between the original and augmented optical flow maps helps the model improve its predictions on unseen data.

Figure 9: Percentage of absolute EPE score changes
Figure 10: EPE scores for different configurations
Table 11: EPE scores for different configurations

Batch Size	Number of Augmentations	EPE Value
8	2	2.75277
2	11	2.75071
6	3	2.78732
2	3	2.74143
6	2	2.80244
2	2	2.76932

Figure 9 illustrates percentage of absolute EPE score decrease for different combinations of number of augmentations, number of epochs, and batch size, while Figure 10 illustrates different combinations of those parameters and their corresponding EPE scores. Table 11 presents EPE scores for different batch sizes and number of augmentations using a system equipped with six GPUs.

Across these configurations, a reduction in EPE is observed, although the trend is not entirely consistent. This suggests that while augmentations and optimizations contribute to minimizing EPE, further tuning of batch size, GPU count, and augmentation strategies is needed to achieve definitive improvements.

For example, the configuration with a batch size of 2 and 11 augmentations resulted in the lowest EPE value of 2.75071. This configuration illustrates how batch size is calculated in distributed training, where the net batch size is determined using: \[ \text{Net Batch Size} = \text{Batch Size} \times (\text{Num of Augmentations} + 1) \times \text{Number of GPUs} \] In this case: \[ 2 \times (11 + 1) \times 6 = 144 \]

Tech Stack

• Framework: Torch Scale
• Framework: PyTorch Lightning Fabric

WanDB Report

Currently in progress and kept private.
Close

Natural Legal Language Processing

Abstract A key component of the Natural Language Processing (NLP) pipeline is Sentence Boundary Detection (SBD). Erroneous SBD could affect other processing steps and reduce performance. A few criteria based on punctuation and capitalization are necessary to identify sentence borders in well-defined corpora. However, due to several grammatical ambiguities, the complex structure of legal data poses difficulties for SBD. In this paper, we have trained a neural network framework for identifying the end of the sentence in legal text. We used several state-of-the-art deep learning models, analyzed their performance, and identified that Convolutional Neural Network(CNN) outperformed other deep learning frameworks. We compared the results with rule-based, statistical, and transformer-based frameworks. The best neural network model outscored the popular rule-based framework with an improvement of 8% in the F1 score. Although domain-specific statistical models have slightly improved performance, the trained CNN is 80 times faster in run-time and doesn’t require much feature engineering. Furthermore, after extensive pretraining, the transformer models fall short in overall performance compared to the best deep learning model.

Close Github

Intelligent OCR

This project involves the development of an advanced Optical Character Recognition (OCR) software, leveraging a combination of cutting-edge neural network models: CRAFT, Faster R-CNN, Tesseract, and a Siamese network model. The software is hosted on Azure Cloud, utilizing a virtual machine (VM) to run the server and process documents. The entire system is designed to convert scanned documents into editable text, extract bounding boxes for words and sentences, classify sentences into categories, and provide a comprehensive annotation interface for user modifications.

Key Features:

• Model Integration: The OCR software integrates multiple state-of-the-art models trained using PyTorch on the FUND dataset. This setup ensures high accuracy and reliability in text extraction and classification.
• Document Processing: Users can upload scanned documents in various formats (.png, .jpg, .jpeg, and .pdf) to the frontend website. For PDFs, only the first page is processed. The software extracts editable text, provides bounding boxes for each word and sentence, and classifies sentences into categories such as 'other', 'question', 'answer', and 'header'.
• Annotation Interface: The frontend features a user-friendly annotation interface built with annotorious.js. Users can modify model predictions or annotate documents from scratch. Annotations can be saved and downloaded in a simple .txt format.
• Offline Processing: The software supports offline batch processing, allowing users to process multiple images at once. Users can choose between output formats like MTX or FUND dataset formats.
• Server Details: The models run on an Azure VM with Linux (Ubuntu 18.04), specifically a Standard B2s instance (2 vCPUs, 4 GiB memory). Due to server connection constraints, there might be occasional interruptions in access, but comprehensive setup instructions are provided for replicating the environment.
• Training and Metrics: Model training is conducted using PyTorch, with detailed explanations of training steps and performance metrics. All trained models and predictions on public test datasets are available for download.

Close Github Demo

i-Pravesh

I-Pravesh is a sophisticated Smart Attendance Android application designed to simplify and enhance the attendance recording process. It uses advanced face detection and recognition technologies, specifically MobileFaceNet combined with TensorFlow Lite, for accurate biometric authentication. The app not only performs fast and reliable face recognition but also ensures that the user is within a designated location through geofencing. It maintains face embeddings and encrypts credentials for secure data handling. Additionally, I-Pravesh features a comprehensive dashboard for both users and administrators, providing a user-friendly interface for managing and monitoring attendance records efficiently.

Close Github

SummarizeIQ: An Integrated Summarization and Content Analysis Engine

SummarizeIQ combines T5 summarization with Torch Serve, incorporating advanced extractive methods like LexRank and TextRank for enhanced text processing. The system features keyword generation and employs query-based content filtering using cosine similarity to deliver precise and relevant summaries. It also boasts a user-friendly website interface, allowing seamless interaction with the summarization tools. Additionally, SummarizeIQ supports multi-threaded web scraping, enabling efficient and concurrent data extraction from various sources for comprehensive article analysis and summarization.

Close Github

Real Estate VR

Problem Statement
Traditional real estate methods rely on physical visits and static images, limiting immersion and convenience. To enhance the real estate experience, we aim to integrate virtual reality (VR) technology, creating a user-friendly, cross-platform VR application. This solution will offer immersive property exploration and real-time updates, modernizing how users view and interact with properties.

Design
1. Asset Coordination and Environment Generation: Utilizing Unity’s assets and spatial mapping, we create a realistic, adaptable virtual environment for immersive exploration.
2. Advanced User Interaction with XR Device Simulator: An intuitive interface with Unity’s XR Device Simulator allows seamless navigation and interaction through a 'digital wand,' enhancing user experience with C# scripting.
3. Dynamic Building Placement with Spatial Intelligence: Real-time building placement is facilitated by Unity’s spatial analysis, offering precise options based on spatial context.
4. Futuristic Navigation Paradigms: Advanced navigation includes teleportation and aerial views, allowing instant spatial traversal and comprehensive property inspection.
5. Precision-Engineered C# Scripting: C# scripts ensure smooth user controls, dynamic building placement, and effective interaction management.
Our VR platform allows users to add plots, select and place buildings, and view properties with detailed information. It supports flythrough and teleportation, with eagle-eye views for broader city and plot planning.

Key Features
- Add and visualize new plots
- Place houses with size and price details
- Immersive VR interface
- Flythrough and teleportation travel
- Multi-plot monitoring with eagle-eye views

Tools Used
- Unity
- C#

Close Github Demo

Yet Another Python Compiler

A Python Compiler, built with Bison and Flex, processes Python code by tokenizing it, parsing it into a syntax tree, and performing semantic checks. It generates intermediate code and applies optimizations like constant folding and dead code elimination. Key features include lexical analysis, syntax and semantic error detection, and code optimization.

Overview
The project focuses on developing a compiler for a custom programming language, implementing various stages of compilation including lexical analysis, parsing, semantic analysis, intermediate code generation, and code optimization.

Tech Stack
Programming Language: Python
Tools: YACC (Yet Another Compiler Compiler), Lex (for lexical analysis)
Data Structures: Symbol tables, parse trees, abstract syntax trees (ASTs)

Features and Contributions

Lexical Analysis:
Token identification and lexical error detection.
Developed token identification and error detection mechanisms.
Created a symbol table to manage identifiers and constants.

Parser:
Syntax declaration, indentation and syntactic error detection.
Implemented syntax rules using YACC for expressions and control flow.
Generated parse trees and abstract syntax trees (ASTs).

Semantic Analysis:
SDD + SDT, Annotated Parse Tree, and Semantic Error detection.
Designed syntax-directed translations to ensure semantic correctness.
Detected semantic errors such as undeclared variables and misuse of reserved identifiers.

Intermediate Code Generation (ICG):
Generated three-address code and quadruples.
Implemented backpatching for jump statements.

Code Optimization:
Basic blocks, DAG, CFG, Induction Variable elimination.
Applied optimization techniques including constant folding, copy propagation, dead code elimination, and peephole optimization.
Enhanced code efficiency through these optimizations.

Report

Lexical Analsysis

Parser

Semantic Analsysis

ICG

Optimization

Close Github

Oct tree based 3D OpenGL Renderer

The Octree-Based 3D OpenGL Renderer, created with PyOpenGL, efficiently renders 3D scenes by using an octree data structure to manage and optimize rendering performance.

Close Github

Sartorius Cell Instance Segmentation

Trained R-CNN, U-Net and Detectron2 using Pytorch to detect and delineate distinct objects of interest in biological images depicting neuronal cell types commonly used in studying neurological disorders

Key findings include:

1. Model Comparison: Detectron2 outperformed U-Net and Mask R-CNN, showcasing superior accuracy and robustness in segmenting complex neuronal cell structures.
2. U-Net: Known for capturing fine details, U-Net was used with a ResNet34 backbone and a combination of focal and dice losses to handle class imbalance. Post-processing improved segmentation outcomes, though some blurriness persisted.
3. Mask R-CNN: Leveraging ResNet50, this model generated object bounding boxes and masks, with loss functions tailored to boundary, mask, and classification losses. Overlapping segmentations were mitigated through pixel-wise intersection operations.
4. Detectron2: As a flexible, high-performance framework, Detectron2 provided modular implementation and incorporated innovative techniques like deformable convolutions, leading to its superior performance in this study.

Close Github

Other Projects

Projects include:
Tumor classification
Pneumonia (COVID-19) chest X-ray classification
Neural style transfer
Crack detection with TF Lite
Face detection using a Siamese Neural Network model to develop a one-shot neural network with FaceNet and MTCNN as the backbone Unity based C# game called "The Keres"

Close