Annibale Panichella | CISELab

Multi-objective differential evolution in the generation of adversarial examples

Thu, 06 Jun 2024 17:56:40 +0200

Danger is My Middle Lane: Simulations from Real-World Dangerous Roads

Thu, 06 Jun 2024 09:16:23 +0200

Higher Fault Detection Through Novel Density Estimators in Unit Test Generation

Thu, 06 Jun 2024 09:16:23 +0200

Evolutionary Generative Fuzzing for Differential Testing of the Kotlin Compiler

Thu, 04 Apr 2024 17:56:40 +0200

Syntest-JavaScript: Automated Unit-Level Test Case Generation for JavaScript

Fri, 02 Feb 2024 17:56:40 +0200

The Slow and The Furious? Performance Antipattern Detection in Cyber-Physical Systems

Mon, 01 Jan 2024 17:32:54 +0200

Breaking the Silence: the Threats of Using LLMs in Software Engineering

Mon, 01 Jan 2024 14:29:50 +0200

Abstract

Large Language Models (LLMs) have gained considerable traction within the Software Engineering (SE) community, impacting various SE tasks from code completion to test generation, from program repair to code summarization. Despite their promise, researchers must still be careful as numerous intricate factors can influence the outcomes of experiments involving LLMs. This paper initiates an open discussion on potential threats to the validity of LLM-based research including issues such as closed-source models, possible data leakage between LLM training data and research evaluation, and the reproducibility of LLM-based findings. In response, this paper proposes a set of guidelines tailored for SE researchers and Language Model (LM) providers to mitigate these concerns. The implications of the guidelines are illustrated using existing good practices followed by LLM providers and a practical example for SE researchers in the context of test case generation.

Multi-objective Black-Box Test Case Prioritization Based on Wordnet Distances

Wed, 18 Oct 2023 09:16:23 +0200

Abstract

Test case prioritization techniques have emerged as effective strategies to optimize this process and mitigate the regression testing costs. Commonly, black-box heuristics guide optimal test ordering, leveraging information retrieval (e.g., cosine distance) to measure the test case distance and sort them accordingly. However, a challenge arises when dealing with tests of varying granularity levels, as they may employ distinct vocabularies (e.g., name identifiers). In this paper, we propose to measure the distance between test cases based on the shortest path between their identifiers within the WordNet lexical database. This additional heuristic is combined with the traditional cosine distance to prioritize test cases in a multi-objective fashion. Our preliminary study conducted with two different Java projects shows that test cases prioritized with WordNet achieve larger fault detection capability (APFD) compared to the traditional cosine distance used in the literature.

A Fast Multi-objective Evolutionary Approach for Designing Large-Scale Optical Mode Sorter

Tue, 25 Apr 2023 10:56:44 +0200

Searching for Quality: Genetic Algorithms and Metamorphic Testing for Software Engineering ML

Tue, 25 Apr 2023 10:56:44 +0200

Evolutionary Approach for Concurrency Testing of Ripple Blockchain Consensus Algorithm

Sun, 01 Jan 2023 22:27:51 +0200

HasBugs - Handpicked Haskell Bugs

Sun, 01 Jan 2023 17:08:37 +0100

Grammar-Based Evolutionary Fuzzing for JSON-RPC APIs

Sun, 01 Jan 2023 17:56:40 +0200

On the Strengths of Pure Evolutionary Algorithms in Generating Adversarial Examples

Sun, 01 Jan 2023 17:56:40 +0200

JUGE: An Infrastructure for Benchmarking Java Unit Test Generators

Mon, 12 Dec 2022 11:11:22 +0200

Continuous Integration and Delivery practices for Cyber- Physical systems: An interview-based study

Fri, 11 Nov 2022 11:21:45 +0200

Generating Class-Level Integration Tests Using Call Site Information

Tue, 13 Sep 2022 09:29:14 +0200

Abstract: Search-based approaches have been used in the literature to automate the process of creating unit test cases. However, related work has shown that generated tests with high code coverage could be ineffective, i.e., they may not detect all faults or kill all injected mutants. In this paper, we propose CLING, an integration-level test case generation approach that exploits how a pair of classes, the caller and the callee, interact with each other through method calls. In particular, CLING generates integration-level test cases that maximize the Coupled Branches Criterion (CBC). Coupled branches are pairs of branches containing a branch of the caller and a branch of the callee such that an integration test that exercises the former also exercises the latter. CBC is a novel integration-level coverage criterion, measuring the degree to which a test suite exercises the interactions between a caller and its callee classes. We implemented CLING and evaluated the approach on 140 pairs of classes from five different open-source Java projects. Our results show that (1) CLING generates test suites with high CBC coverage, thanks to the definition of the test suite generation as a many-objectives problem where each couple of branches is an independent objective; (2) such generated suites trigger different class interactions and can kill on average 7.7% (with a maximum of 50%) of mutants that are not detected by tests generated randomly or at the unit level; (3) CLING can detect integration faults coming from wrong assumptions about the usage of the callee class (25 for our subject systems) that remain undetected when using automatically generated random and unit-level test suites.

Guess What: Test Case Generation for Javascript with Unsupervised Probabilistic Type Inference

Thu, 18 Aug 2022 09:16:23 +0200

Abstract

Search-based test case generation approaches make use of static type information to determine which data types should be used for the creation of new test cases. Dynamically typed languages like JavaScript, however, do not have this type information. In this paper, we propose an unsupervised probabilistic type inference approach to infer data types within the test case generation process. We evaluated the proposed approach on a benchmark of 98~units under test (i.e., exported classes and functions) compared to random type sampling w.r.t. branch coverage. Our results show that our type inference approach achieves a statistically significant increase in 56% of the test files with up to 71% of branch coverage compared to the baseline.

Test Smells 20 Years Later: Detectability, Validity, and Reliability

Thu, 21 Jul 2022 11:34:14 +0200

Guiding Automated Test Case Generation for Transaction-Reverting Statements in Smart Contracts

Sat, 11 Jun 2022 18:43:48 +0200

Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments

Mon, 04 Apr 2022 11:21:45 +0200

An Improved Pareto Front Modeling Algorithm for Large-scale Many-Objective Optimization

Sun, 27 Mar 2022 10:56:44 +0200

SynTest-Solidity: Automated Test Case Generation and Fuzzing for Smart Contracts

Tue, 18 Jan 2022 14:29:50 +0200

Large scale inverse design of planar on-chip mode sorter

Sat, 01 Jan 2022 09:57:31 +0100

PropR: Property-Based Automatic Program Repair

Sat, 01 Jan 2022 09:32:39 +0100

Assessing Robustness of ML-Based Program Analysis Tools using Metamorphic Program Transformations

Sat, 07 Aug 2021 22:27:51 +0200

Improving Test Case Generation for REST APIs Through Hierarchical Clustering

Wed, 07 Jul 2021 22:27:51 +0200

Hybrid Multi-level Crossover for Unit Test Case Generation

Tue, 29 Jun 2021 09:16:23 +0200

Abstract

State-of-the-art search-based approaches for test case generation work at test case level, where tests are represented as sequences of statements. These approaches make use of genetic operators (i.e., mutation and crossover) that create test variants by adding, altering, and removing statements from existing tests. While this encoding schema has been shown to be very effective for many-objective test case generation, the standard crossover operator (single-point) only alters the structure of the test cases but not the input data. In this paper, we argue that changing both the test case structure and the input data is necessary to increase the genetic variation and improve the search process. Hence, we propose a hybrid multi-level crossover (HMX) operator that combines the traditional test-level crossover with data-level recombination. The former evolves and alters the test case structures, while the latter evolves the input data using numeric and string-based recombinational operators. We evaluate our new crossover operator by performing an empirical study on more than 100 classes selected from open-source Java libraries for numerical operations and string manipulation. We compare HMX with the single-point crossover that is used in EvoSuite w.r.t structural coverage and fault detection capability. Our results show that HMX achieves a statistically significant increase in 30% of the classes up to 19% in structural coverage compared to the single-point crossover. Moreover, the fault detection capability improved up to 12% measured using strong mutation score.

Multi-objective Test Case Selection Through Linkage Learning-driven Crossover

Tue, 29 Jun 2021 09:16:23 +0200

Abstract

Test case selection (TCS) aims to select a subset of the test suite to run for regression testing. The selection is typically based on past coverage and execution cost data. Researchers have successfully used multi-objective evolutionary algorithms (MOEAs), such as NSGA-II and its variants, to solve the problem. These MOEAs use traditional crossovers to create new candidate solutions during the search. Recent studies in evolutionary computation showed that more effective recombinations can be made by using linkage learning. Inspired by these recent advances in this field, we propose a new variant of NSGA-II, called L2-NSGA, that uses linkage learning to optimize test case selection. In particular, we use an unsupervised clustering algorithm to infer promising patterns among the solutions (sub-test suites). Then, these patterns are used in the next iterations of L2-NSGA to create solutions that contain/preserve these inferred patterns. Our results show that our customizations make NSGA-II more effective for test case selection. Furthermore, the test suite sub-sets generated by L2-NSGA are less expensive and more effective (detect more faults) than those generated by MOEAs used in the literature for regression testing.

Log-based Slicing for System-level Test Cases

Mon, 19 Apr 2021 09:03:56 +0200

What Are We Really Testing in Mutation Testing for Machine Learning? A Critical Reflection

Mon, 18 Jan 2021 14:29:50 +0200

Search-Based Software Re-Modularization: A Case Study at Adyen

Fri, 01 Jan 2021 22:27:51 +0200

How to Kill Them All: An Exploratory Study on the Impact of Code Observability on Mutation Testing

Tue, 01 Dec 2020 17:32:54 +0200

Serverless Testing: Tool Vendors' and Experts' Point of View

Wed, 07 Oct 2020 10:14:08 +0200

A Systematic Comparison of Search-Based Approaches for LDA Hyperparameter Tuning

Wed, 09 Sep 2020 21:50:29 +0200

Revisiting Test Smells in Automatically Generated Tests: Limitations, Pitfalls, and Opportunities

Tue, 04 Aug 2020 16:05:57 +0200

DeepTC-Enhancer: Improving the Readability of Automatically Generated Tests

Thu, 30 Jul 2020 22:27:51 +0200

Generating Highly-structured Input Data by Combining Search-based Testing and Grammar-based Fuzzing

Mon, 06 Jul 2020 14:29:50 +0200

Botsing, a Search-based Crash Reproduction Framework for Java

Sun, 05 Jul 2020 14:19:53 +0200

Crash Reproduction Using Helper Objectives

Sat, 04 Jul 2020 18:56:54 +0200

Automated Repair of Feature Interaction Failures in Automated Driving Systems

Fri, 17 Apr 2020 17:32:54 +0200

LogChunks: A Data Set for Build Log Analysis

Tue, 03 Mar 2020 17:08:37 +0100

Testing with Fewer Resources: An Adaptive Approach to Performance-Aware Test Case Generation

Wed, 09 Oct 2019 09:51:33 +0200

Abstract: Automated test case generation is an effective technique to yield high-coverage test suites. While the majority of research effort has been devoted to satisfying coverage criteria, a recent trend emerged towards optimizing other non-coverage aspects. In this regard, runtime and memory usage are two essential dimensions: less expensive tests reduce the resource demands for the generation process and later regression testing phases. This study shows that performance-aware test case generation requires solving two main challenges: providing a good approximation of resource usage with minimal overhead and
avoiding detrimental effects on both final coverage and fault detection effectiveness. To tackle these challenges, we conceived a set of performance proxies -inspired by previous work on performance testing- that provide a reasonable estimation of the test execution costs (i.e., runtime and memory usage). Thus, we propose an adaptive strategy, called aDynaMOSA, which leverages these proxies by extending DynaMOSA, a state-of-the-art evolutionary algorithm in unit testing. Our empirical study -involving 110 non-trivial Java classes- reveals that our adaptive approach generates test suite with statistically significant improvements in runtime (-25%) and heap memory consumption (-15%) compared to DynaMOSA. Additionally, aDynaMOSA has comparable results to DynaMOSA over seven different coverage criteria and similar fault detection effectiveness. Our empirical investigation also highlights that the usage of performance proxies (i.e., without the adaptiveness) is not sufficient to generate more performant test cases without compromising the overall coverage.

JCOMIX: A Search-Based Tool to Detect XML Injection Vulnerabilities in Web Applications

Wed, 26 Jun 2019 10:41:41 +0200

Effective and Efficient API Misuse Detection via Exception Propagation and Search-based Testing

Wed, 01 May 2019 20:22:51 +0200

Abstract : Application Programming Interfaces (APIs) typically come with (implicit) usage constraints. The violations of these constraints (API misuses) can lead to software crashes. Even though there are several tools that can detect API misuses, most of them suffer from a very high rate of false positives. We introduce Catcher, a novel API misuse detection approach that combines static exception propagation analysis with automatic search-based test case generation to effectively and efficiently pinpoint crash-prone API misuses in client applications. We validate Catcher against 21 Java applications, targeting misuses of the Java platform’s API. Our results indicate that Catcher is able to generate test cases that uncover 243 (unique) API misuses that result in crashes. Our empirical evaluation shows that Catcher can detect a large number of misuses (77 cases) that would remain undetected by the traditional coverage-based test case generator EvoSuite. Additionally, Catcher is on average eight times faster than EvoSuite in generating test cases for the identified misuses. Finally, we find that the majority of the exceptions triggered by Catcher are unexpected to developers i.e., not only unhandled in the source code but also not listed in the documentation of the client applications.

A Systematic Comparison of Search Algorithms for Topic Modelling - A Study on Duplicate Bug Report Identification

Mon, 29 Apr 2019 23:16:23 +0200

Abstract

Latent Dirichlet Allocation (LDA) has been used to support many software engineering tasks. Previous studies showed that default settings lead to sub-optimal topic modeling with a dramatic impact on the performance of such approaches in terms of precision and recall. For this reason, researchers used search algorithms (e.g., genetic algorithms) to automatically configure topic models in an unsupervised fashion. While previous work showed the ability of individual search algorithms in finding near-optimal configurations, it is not clear to what extent the choice of the meta-heuristic matters for SE tasks. In this paper, we present a systematic comparison of five different meta-heuristics to configure LDA in the context of duplicate bug reports identification. The results show that (1) no master algorithm outperforms the others for all software projects, (2) random search and PSO are the least effective meta-heuristics. Finally, the running time strongly depends on the computational complexity of LDA while the internal complexity of the search algorithms plays a negligible role.

An Adaptive Evolutionary Algorithm based on Non-Euclidean Geometry for Many-objective Optimization

Mon, 08 Apr 2019 10:45:30 +0200

EvoSuite at the SBST 2019 Tool Competition

Mon, 08 Apr 2019 10:41:50 +0200

Beyond Unit-Testing in Search-based Test Case Generation: Challenges and Opportunities

Mon, 08 Apr 2019 10:39:03 +0200

Search-based Multi-Vulnerability Testing of XML Injections in Web Applications

Mon, 08 Apr 2019 10:11:19 +0200

AI for Fintech Research

Tue, 12 Mar 2019 00:00:00 +0000

AI for Fintech Research (AFR) is a collaboration between ING and Delft University of Technology. The mission of AFR is to perform world-class research at the intersection of Artificial Intelligence, Data Analytics, and Software Analytics in the context of FinTech. AFR was launched in 2020, and will run for a period of five years.

With 36 million customers, activities in 42 countries, and a total of 50,000 employees of which 15,000 work in IT, software and data technology is at the heart of ING’s business and operations. In this context, AFR seeks to develop new AI-driven theories, methods, and tools in large scale data and software analytics.

The core of the AFR consists of 10 research tracks, in which 10 PhD students work on a range of topics, such as software analytics, data integration, fairness in machine learning, model life cycle management, regulatory compliance, search-based software engineering, and A/B testing. In each track, researchers from TU Delft and engineers from ING will be involved, as well as bachelor and master students from TU Delft.

AFR is bi-located at the TU Delft campus in Delft and at Cumulus Park – the collaborative innovation district in Amsterdam Southeast – bringing together students, engineers, researchers, professors, and entrepreneurs from both organizations at both locations.

AFR participates in the Innovation Center for Artificial Intelligence (ICAI) as one of its labs. ICAI is a virtual organization consisting of a series of labs of similar size (over five PhD researchers each) funded directly by industry. AFR directly benefits from the experience and expertise of other academic and industrial ICAI partners, such as Qualcomm, Bosch, Ahold Delhaize, the Dutch National Police, the University of Amsterdam, and Vrije Universiteit.

The scientific director of AFR is Arie van Deursen, professor in the Software Engineering Research Group (SERG) at Delft University of Technology.

A Test Case Prioritization Genetic Algorithm guided by the Hypervolume Indicator

Sat, 08 Sep 2018 11:51:02 +0200

A Large Scale Empirical Comparison of State-of-the-art Search-based Test Case Generators

Sat, 18 Aug 2018 07:39:53 +0200

Search-Based Crash Reproduction and Its Impact on Debugging

Mon, 13 Aug 2018 18:55:36 +0200

Incremental Control Dependency Frontier Exploration for Many-Criteria Test Case Generation

Fri, 20 Jul 2018 00:07:54 +0200

Single-objective versus Multi-Objectivized Optimization for Evolutionary Crash Reproduction

Fri, 20 Jul 2018 00:06:34 +0200

A Search-based Approach for Accurate Identification of Log Message Formats

Fri, 20 Jul 2018 00:05:13 +0200

Search-Based Test Data Generation for SQL Queries

Fri, 20 Jul 2018 00:04:04 +0200

An Investigation of Compression Techniques to Speed up Mutation Testing

Fri, 20 Jul 2018 00:03:02 +0200

Java Unit Testing Tool Competition - Sixth Round

Fri, 20 Jul 2018 00:01:14 +0200

The Scent of a Smell: An Extensive Comparison between Textual and Structural Smells

Tue, 10 Jul 2018 11:25:49 +0200

Developer Testing in The IDE: Patterns, Beliefs, And Behavior

Tue, 10 Jul 2018 11:24:46 +0200

Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets

Tue, 10 Jul 2018 11:23:34 +0200

Automatic Generation of Tests to Exploit XML Injection Vulnerabilities in Web Applications

Tue, 10 Jul 2018 11:21:45 +0200

A Machine Learning- Driven Evolutionary Approach for Testing Web Application Firewalls

Tue, 10 Jul 2018 11:11:22 +0200

DevOps for Complex Cyber-physical Systems (COSMOS)

Mon, 01 Jan 0001 00:00:00 +0000

Emerging Cyber-Physical Systems (CPS)—from robotics, transportation, to medical devices—play a crucial role in the quality of life of European citizens and the future of the European economy. Increasing automation to such an extent, however, gives rise to many challenges, at the crux of which lies the hardware and software symbiosis. COSMOS proposes to overcome the strain on developing and evolving high-quality, dependable CPS by employing two key technologies: DevOps and Artificial Intelligence (AI). These technologies offer the potential to address CPS development, verification, and evolution.

SynTest

Mon, 01 Jan 0001 00:00:00 +0000

A framework for generating synthetic tests for programs based on JavaScript

University Blockchain Research Initiative

Mon, 01 Jan 0001 00:00:00 +0000

UBRI is a partnership between Ripple and top universities around the world to support academic research, technical development and innovation in blockchain, cryptocurrency, and digital payments. Ripple is providing both financial and technical resources to university partners and collaborates with faculty and students on research and technical projects.