1 | CISELab

Danger is My Middle Lane: Simulations from Real-World Dangerous Roads

Thu, 06 Jun 2024 09:16:23 +0200

Higher Fault Detection Through Novel Density Estimators in Unit Test Generation

Thu, 06 Jun 2024 09:16:23 +0200

Evolutionary Generative Fuzzing for Differential Testing of the Kotlin Compiler

Thu, 04 Apr 2024 17:56:40 +0200

Syntest-JavaScript: Automated Unit-Level Test Case Generation for JavaScript

Fri, 02 Feb 2024 17:56:40 +0200

CATMA: Conformance Analysis Tool For Microservice Applications

Mon, 01 Jan 2024 19:37:11 +0200

Abstract:

The microservice architecture allows developers to divide the core functionality of their software system into multiple smaller services. However, this architectural style also makes it harder for them to debug and assess whether the system’s deployment conforms to its implementation. We present CATMA, an automated tool that detects non-conformances between the system’s deployment and implementation. It automatically visualizes and generates potential interpretations for the detected discrepancies. Our evaluation of CATMA shows promising results in terms of performance and providing useful insights. CATMA is available at https://cyber-analytics.nl/catma.github.io/, and a demonstration video is available at https://youtu.be/WKP1hG-TDKc.

TestSpark: IntelliJ IDEA’s Ultimate Test Generation Companion

Mon, 01 Jan 2024 19:37:11 +0200

Abstract:

Writing software tests is laborious and time-consuming. To address this, prior studies introduced various automated test-generation techniques. A well-explored research direction in this field is unit test generation, wherein artificial intelligence (AI) techniques create tests for a method/class under test. While many of these techniques have primarily found applications in a research context, existing tools (e.g., EvoSuite, Randoop, and AthenaTest) are not user-friendly and are tailored to a single technique. This paper introduces Test- Spark, a plugin for IntelliJ IDEA that enables users to generate unit tests with only a few clicks directly within their Integrated De- velopment Environment (IDE). Furthermore, TestSpark also allows users to easily modify and run each generated test and integrate them into the project workflow. TestSpark leverages the advances of search-based test generation tools, and it introduces a technique to generate unit tests using Large Language Models (LLMs) by creating a feedback cycle between the IDE and the LLM. Since TestSpark is an open-source (https://github.com/JetBrains-Research/TestSpark), extendable, and well-documented tool, it is possible to add new test generation methods into the plugin with the minimum effort. This paper also explains our future studies related to TestSpark and our preliminary results.

Breaking the Silence: the Threats of Using LLMs in Software Engineering

Mon, 01 Jan 2024 14:29:50 +0200

Abstract

Large Language Models (LLMs) have gained considerable traction within the Software Engineering (SE) community, impacting various SE tasks from code completion to test generation, from program repair to code summarization. Despite their promise, researchers must still be careful as numerous intricate factors can influence the outcomes of experiments involving LLMs. This paper initiates an open discussion on potential threats to the validity of LLM-based research including issues such as closed-source models, possible data leakage between LLM training data and research evaluation, and the reproducibility of LLM-based findings. In response, this paper proposes a set of guidelines tailored for SE researchers and Language Model (LM) providers to mitigate these concerns. The implications of the guidelines are illustrated using existing good practices followed by LLM providers and a practical example for SE researchers in the context of test case generation.

Multi-objective Black-Box Test Case Prioritization Based on Wordnet Distances

Wed, 18 Oct 2023 09:16:23 +0200

Abstract

Test case prioritization techniques have emerged as effective strategies to optimize this process and mitigate the regression testing costs. Commonly, black-box heuristics guide optimal test ordering, leveraging information retrieval (e.g., cosine distance) to measure the test case distance and sort them accordingly. However, a challenge arises when dealing with tests of varying granularity levels, as they may employ distinct vocabularies (e.g., name identifiers). In this paper, we propose to measure the distance between test cases based on the shortest path between their identifiers within the WordNet lexical database. This additional heuristic is combined with the traditional cosine distance to prioritize test cases in a multi-objective fashion. Our preliminary study conducted with two different Java projects shows that test cases prioritized with WordNet achieve larger fault detection capability (APFD) compared to the traditional cosine distance used in the literature.

A Fast Multi-objective Evolutionary Approach for Designing Large-Scale Optical Mode Sorter

Tue, 25 Apr 2023 10:56:44 +0200

Searching for Quality: Genetic Algorithms and Metamorphic Testing for Software Engineering ML

Tue, 25 Apr 2023 10:56:44 +0200

Evolutionary Approach for Concurrency Testing of Ripple Blockchain Consensus Algorithm

Sun, 01 Jan 2023 22:27:51 +0200

HasBugs - Handpicked Haskell Bugs

Sun, 01 Jan 2023 17:08:37 +0100

Grammar-Based Evolutionary Fuzzing for JSON-RPC APIs

Sun, 01 Jan 2023 17:56:40 +0200

On the Strengths of Pure Evolutionary Algorithms in Generating Adversarial Examples

Sun, 01 Jan 2023 17:56:40 +0200

Guess What: Test Case Generation for Javascript with Unsupervised Probabilistic Type Inference

Thu, 18 Aug 2022 09:16:23 +0200

Abstract

Search-based test case generation approaches make use of static type information to determine which data types should be used for the creation of new test cases. Dynamically typed languages like JavaScript, however, do not have this type information. In this paper, we propose an unsupervised probabilistic type inference approach to infer data types within the test case generation process. We evaluated the proposed approach on a benchmark of 98~units under test (i.e., exported classes and functions) compared to random type sampling w.r.t. branch coverage. Our results show that our type inference approach achieves a statistically significant increase in 56% of the test files with up to 71% of branch coverage compared to the baseline.

Guiding Automated Test Case Generation for Transaction-Reverting Statements in Smart Contracts

Sat, 11 Jun 2022 18:43:48 +0200

An Improved Pareto Front Modeling Algorithm for Large-scale Many-Objective Optimization

Sun, 27 Mar 2022 10:56:44 +0200

SynTest-Solidity: Automated Test Case Generation and Fuzzing for Smart Contracts

Tue, 18 Jan 2022 14:29:50 +0200

PropR: Property-Based Automatic Program Repair

Sat, 01 Jan 2022 09:32:39 +0100

Assessing Robustness of ML-Based Program Analysis Tools using Metamorphic Program Transformations

Sat, 07 Aug 2021 22:27:51 +0200

Improving Test Case Generation for REST APIs Through Hierarchical Clustering

Wed, 07 Jul 2021 22:27:51 +0200

Hybrid Multi-level Crossover for Unit Test Case Generation

Tue, 29 Jun 2021 09:16:23 +0200

Abstract

State-of-the-art search-based approaches for test case generation work at test case level, where tests are represented as sequences of statements. These approaches make use of genetic operators (i.e., mutation and crossover) that create test variants by adding, altering, and removing statements from existing tests. While this encoding schema has been shown to be very effective for many-objective test case generation, the standard crossover operator (single-point) only alters the structure of the test cases but not the input data. In this paper, we argue that changing both the test case structure and the input data is necessary to increase the genetic variation and improve the search process. Hence, we propose a hybrid multi-level crossover (HMX) operator that combines the traditional test-level crossover with data-level recombination. The former evolves and alters the test case structures, while the latter evolves the input data using numeric and string-based recombinational operators. We evaluate our new crossover operator by performing an empirical study on more than 100 classes selected from open-source Java libraries for numerical operations and string manipulation. We compare HMX with the single-point crossover that is used in EvoSuite w.r.t structural coverage and fault detection capability. Our results show that HMX achieves a statistically significant increase in 30% of the classes up to 19% in structural coverage compared to the single-point crossover. Moreover, the fault detection capability improved up to 12% measured using strong mutation score.

Multi-objective Test Case Selection Through Linkage Learning-driven Crossover

Tue, 29 Jun 2021 09:16:23 +0200

Abstract

Test case selection (TCS) aims to select a subset of the test suite to run for regression testing. The selection is typically based on past coverage and execution cost data. Researchers have successfully used multi-objective evolutionary algorithms (MOEAs), such as NSGA-II and its variants, to solve the problem. These MOEAs use traditional crossovers to create new candidate solutions during the search. Recent studies in evolutionary computation showed that more effective recombinations can be made by using linkage learning. Inspired by these recent advances in this field, we propose a new variant of NSGA-II, called L2-NSGA, that uses linkage learning to optimize test case selection. In particular, we use an unsupervised clustering algorithm to infer promising patterns among the solutions (sub-test suites). Then, these patterns are used in the next iterations of L2-NSGA to create solutions that contain/preserve these inferred patterns. Our results show that our customizations make NSGA-II more effective for test case selection. Furthermore, the test suite sub-sets generated by L2-NSGA are less expensive and more effective (detect more faults) than those generated by MOEAs used in the literature for regression testing.

Log-based Slicing for System-level Test Cases

Mon, 19 Apr 2021 09:03:56 +0200

EvoSuite at the SBST 2021 Tool Competition

Thu, 01 Apr 2021 17:56:40 +0200

What Are We Really Testing in Mutation Testing for Machine Learning? A Critical Reflection

Mon, 18 Jan 2021 14:29:50 +0200

Search-Based Software Re-Modularization: A Case Study at Adyen

Fri, 01 Jan 2021 22:27:51 +0200

ReproducedPaper.org: Openly teaching and structuring machine learning reproducibility

Fri, 01 Jan 2021 17:32:54 +0200

Revisiting Test Smells in Automatically Generated Tests: Limitations, Pitfalls, and Opportunities

Tue, 04 Aug 2020 16:05:57 +0200

DeepTC-Enhancer: Improving the Readability of Automatically Generated Tests

Thu, 30 Jul 2020 22:27:51 +0200

Good Things Come In Threes: Improving Search-based Crash Reproduction With Helper Objectives

Thu, 30 Jul 2020 22:27:46 +0200

Generating Highly-structured Input Data by Combining Search-based Testing and Grammar-based Fuzzing

Mon, 06 Jul 2020 14:29:50 +0200

Botsing, a Search-based Crash Reproduction Framework for Java

Sun, 05 Jul 2020 14:19:53 +0200

Crash Reproduction Using Helper Objectives

Sat, 04 Jul 2020 18:56:54 +0200

EvoSuite at the SBST 2020 Tool Competition

Fri, 17 Apr 2020 17:56:40 +0200

Automated Repair of Feature Interaction Failures in Automated Driving Systems

Fri, 17 Apr 2020 17:32:54 +0200

LogChunks: A Data Set for Build Log Analysis

Tue, 03 Mar 2020 17:08:37 +0100

JCOMIX: A Search-Based Tool to Detect XML Injection Vulnerabilities in Web Applications

Wed, 26 Jun 2019 10:41:41 +0200

Effective and Efficient API Misuse Detection via Exception Propagation and Search-based Testing

Wed, 01 May 2019 20:22:51 +0200

Abstract : Application Programming Interfaces (APIs) typically come with (implicit) usage constraints. The violations of these constraints (API misuses) can lead to software crashes. Even though there are several tools that can detect API misuses, most of them suffer from a very high rate of false positives. We introduce Catcher, a novel API misuse detection approach that combines static exception propagation analysis with automatic search-based test case generation to effectively and efficiently pinpoint crash-prone API misuses in client applications. We validate Catcher against 21 Java applications, targeting misuses of the Java platform’s API. Our results indicate that Catcher is able to generate test cases that uncover 243 (unique) API misuses that result in crashes. Our empirical evaluation shows that Catcher can detect a large number of misuses (77 cases) that would remain undetected by the traditional coverage-based test case generator EvoSuite. Additionally, Catcher is on average eight times faster than EvoSuite in generating test cases for the identified misuses. Finally, we find that the majority of the exceptions triggered by Catcher are unexpected to developers i.e., not only unhandled in the source code but also not listed in the documentation of the client applications.

A Systematic Comparison of Search Algorithms for Topic Modelling - A Study on Duplicate Bug Report Identification

Mon, 29 Apr 2019 23:16:23 +0200

Abstract

Latent Dirichlet Allocation (LDA) has been used to support many software engineering tasks. Previous studies showed that default settings lead to sub-optimal topic modeling with a dramatic impact on the performance of such approaches in terms of precision and recall. For this reason, researchers used search algorithms (e.g., genetic algorithms) to automatically configure topic models in an unsupervised fashion. While previous work showed the ability of individual search algorithms in finding near-optimal configurations, it is not clear to what extent the choice of the meta-heuristic matters for SE tasks. In this paper, we present a systematic comparison of five different meta-heuristics to configure LDA in the context of duplicate bug reports identification. The results show that (1) no master algorithm outperforms the others for all software projects, (2) random search and PSO are the least effective meta-heuristics. Finally, the running time strongly depends on the computational complexity of LDA while the internal complexity of the search algorithms plays a negligible role.

An Adaptive Evolutionary Algorithm based on Non-Euclidean Geometry for Many-objective Optimization

Mon, 08 Apr 2019 10:45:30 +0200

EvoSuite at the SBST 2019 Tool Competition

Mon, 08 Apr 2019 10:41:50 +0200

Beyond Unit-Testing in Search-based Test Case Generation: Challenges and Opportunities

Mon, 08 Apr 2019 10:39:03 +0200

Incremental Control Dependency Frontier Exploration for Many-Criteria Test Case Generation

Fri, 20 Jul 2018 00:07:54 +0200

Single-objective versus Multi-Objectivized Optimization for Evolutionary Crash Reproduction

Fri, 20 Jul 2018 00:06:34 +0200

A Search-based Approach for Accurate Identification of Log Message Formats

Fri, 20 Jul 2018 00:05:13 +0200

Search-Based Test Data Generation for SQL Queries

Fri, 20 Jul 2018 00:04:04 +0200

An Investigation of Compression Techniques to Speed up Mutation Testing

Fri, 20 Jul 2018 00:03:02 +0200

Testing Autonomous Cars for Feature Interaction Failures using Many-Objective Search

Fri, 20 Jul 2018 00:01:56 +0200

Java Unit Testing Tool Competition - Sixth Round

Fri, 20 Jul 2018 00:01:14 +0200