Evaluation of local llms for network traffic packet analysis

Sara-Päivi Paukkeri, Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland

Tapio Frantti, Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland

I. Introduction

The number of Internet of Things (IoT) devices has rapidly increased over the years, and this growth is not stopping. In 2024, the number of IoT-connected devices worldwide was approximately 18 billion and is forecasted to reach more than 32 billion in the year 2030 [1]. The already massive amount of data transaction is anticipated to also increase.

However, one of the most significant issues regarding IoT data is its heterogeneity. In real life, an IoT network may have a massive, and probably changing, number of devices. These devices use various protocols, come from different vendors, and have different purposes and security properties. This causes challenges when building intrusion detection systems (IDSs) to protect IoT networks. Developing a monitoring system to detect malicious activities may require significant professional effort, including data preprocessing, labelling training and testing data, and training machine learning (ML) models. In addition, ML models are usually trained and tested with only one or a few datasets in academic research. Thus, even if a trained ML model works well with one dataset, it may not be suitable for real-life solutions. Nevertheless, the field of application for machine learning models ranges from prioritising app reviews [2] to healthcare applications [3] and network security [4].

In 2024, the total cost of data breaches was higher than ever before [5]. In 2023, 54% of organisations were a target of at least one IoT cyberattack attempt every week, on average [6]. This highlights the need to research new methods for addressing the challenges with multisource heterogeneous IoT data.

Several recent works have used large language models (LLMs) in various cybersecurity-related tasks. These include writing malicious code [7–9], hacking websites [10], serving as sparring partners for penetration testers [11], identifying complex vulnerabilities [12], pretending to be a honeypot [13–15], and detecting anomalies in computational workflows [16]. These works have demonstrated that LLMs can understand the field of cybersecurity, at least to some level. Considering IoT network data, one of the most interesting abilities of LLMs is its ability to understand text data and the context of the data. These findings led us to investigate how LLMs can enhance cybersecurity in IoT networks even if they are not initially designed for these tasks. The ability to handle unstructured data may also signal an ability to handle heterogeneous network traffic data.

Ideally, in a star topology-based system, a well-performing LLM-based intrusion detector could be installed on a central device to monitor traffic data. If the LLM detects an attack early enough, it can warn IoT devices about the kind of malicious traffic coming. It may also be possible to have a conversation between IoT devices and an LLM if the IoT devices have sufficient capability, such as an Internet connection. The IoT device could send suspicious packets to the LLM, ask it to analyse them, and deliver possible indicators of attacks (IoAs) to neighbouring devices. However, the essential first step is to examine the strengths and weaknesses of recent LLMs in detecting cyberattacks from IoT traffic data.

This research includes both an in-depth analysis of Llama 3.1:70B as a benchmark and a comparison of five local LLMs. The analysis of Llama 3.1 focused on the ability of the model to analyse packet-level IoT network traffic data. We carefully examined responses of Llama 3.1 to identify the reasons behind successful and unsuccessful answers. The focus areas of our research were the signs of malicious activity that LLMs can detect, the ability of the model to respond rationally, and current limitations of LLMs in analysing network traffic. The LLM comparison investigates whether the main findings from the Llama 3.1 analysis are applicable to other local open-source LLMs. This research is essential for improving the quality of LLM responses and avoiding potential pitfalls in developing real-life solutions and planning future research directions.

The remainder of this paper is organised as follows: Section II presents the related works. Section III details the research process of the Llama 3.1 analysis, including the prompt engineering, data, model selection, results, and discussion. Section IV presents the research process and findings of the LLM comparison. In Section V, we discuss the overall results, followed by Section VI, which outlines the limitations of this study and the directions for future research. Finally, Section VII concludes the paper.

II. Related works

This section provides an overview of related IDS studies on detecting cyberattacks from IoT networks. Makhdoomi et al [17]. and Saif et al [18]. compared machine learning models to detect intrusions in IoT networks. Several other recent works have explored the use of LLMs in cybersecurity, examining their applications for both defensive and offensive purposes. However, detecting malicious activities from heterogeneous, packet-level IoT network traffic using LLMs has received limited attention. Therefore, we focused on LLM-based approaches. We also highlight key solutions proposed to address the challenges of heterogeneous IoT data in IDSs.

In [19], the authors compared GPT-4 and Llama3-8b-Instruct models against traditional ML architectures and transformer models in detecting malicious network traffic flows. They experimented with both zero-shot and fine-tuned models. Their results showed that the LLMs used achieved performance only around random selection, and fine-tuning improved the results only slightly. The authors also tested the ability of LLMs to explain predictions. However, their approach lacks a focus on the reasons behind poor performances and the differences between various attack types.

LLMs have also been used as part of IDSs, while other techniques performed the actual attack detection. The authors in [20] proposed an LLM-enhanced security framework for network traffic data. In their approach, intrusion detection was performed using an isolation forest, while the LLM was used to analyse only statistical features and the anomaly percentage. Thus, they did not evaluate the capabilities of LLMs in analysing packet-level network traffic data. Baral et al [21]. prompted LLMs with information about the attack type detected by an ML model and the most important features identified by explainable AI (XAI) models. The task of LLMs was to explain these features and propose a mitigation plan. In [22], a combination of LLMs and long short-term memory (LSTM) models was used for cyberattack prediction in IoT networks. The authors fine-tuned GPT-2 to predict subsequent packets in network traffic, used a bidirectional encoder representations from transformers (BERT) -model to evaluate the packet predictions and employed an LSTM model to identify malicious packets. In addition [23], evaluated LLMs for explaining the detected anomalies from volumetric network time series data.

Ferrag et al [24]. and Adjewa et al [25]. proposed the use of a BERT-based model with a privacy-preserving fixed-length encoding (PPFLE) method in detecting threats from IoT network traffic data. Ferrag et al [24]. developed an architecture called SecurityBERT for cyber threat detection in IoT networks. They also introduced the PPFLE approach to convert unstructured network traffic data to a structured format closer to the English language to enable BERT models to understand the data better. Adjewa et al [25]. presented a framework for continuous detection, identification, and classification of attacks from network traffic using CSE-CIC-IDS2018 dataset. In addition to a BERT-based model and the PPFLE method, they also used the Gaussian mixture model (GMM) to enable dynamic updates. The framework achieved classification accuracy and recall rates of 95.6% mainly within distributed denial-of-service (DDoS)/DoS and brute force attacks. These frameworks successfully used BERT-based models but did not analyse the current capabilities of the most recent LLMs.

The effects of fine-tuning and context learning for LLMs have also been studied recently. In [26], the authors introduced a framework for LLM-based intrusion detection for wireless Industrial IoT (IIoT) communication. LLMs had four tasks: feature selection, data collection and processing, prompt building, and decision extraction. The authors designed and compared three in-context learning methods for GTP-4 and GPT-3.5: illustrative, heuristic, and interactive in-context learning. However, their data contained only DDoS attacks. Jin et al [16]. experimented with fine-tuned and in-context learned LLMs to evaluate their ability to generalise to other datasets. However, the dataset domain was outside of the IoT network traffic.

An important aspect of detecting cyberattacks from IoT networks is managing the heterogeneity of network traffic data. Multiple methods are proposed for IDSs of IoT networks, but the heterogeneity of multisource IoT data is still a significant problem [27]. Federated learning is one commonly proposed solution to address the challenge posed by heterogeneous IoT data. For example [28], introduces a three-layered hierarchical framework (device, edge server, and cloud) for deploying a federated generative model. The framework also overcomes privacy concerns and communication costs by eliminating the need to send data to a central server. Various solutions focus on data preprocessing to enable the use of traditional ML models in IDSs with heterogeneous IoT data. For example [29], proposed a multiple-input auto-encoder (MIAE) to transform heterogeneous IDS input data into a lower-dimensional representation, which is suitable for traditional classifiers, such as random forest (RF), and [30] used the word embedding (WE) method to eliminate the preprocessing steps of traditional ML to tackle the challenges of heterogeneous data. In addition, Kumar et al. [31], adopted a black hole algorithm for efficient and scalable feature selection for an IDS in IoT networks. However, the abovementioned studies did not use the abilities of LLMs in their solutions.

None of the previous studies deeply analysed the strengths and weaknesses of LLMs in detecting malicious activities from heterogeneous IoT network traffic. As far as we know, this is the first time the ability of an LLM to understand packet-level IoT network traffic data and various cyberattacks was carefully analysed.

III. Llama 3.1 analysis

A. Research process

This research evaluated the effectiveness of LLMs in detecting common cyberattacks from single IoT network traffic packets. Figure 1 illustrates the research workflow in Llama 3.1 analysis. The dataset included 12 attack types in PCAP format, namely port scanning, vulnerability scanning, DDoS, OS fingerprinting, DNS spoofing, ARP spoofing, cross-site scripting (XSS), structured query language (SQL) injection, uploading, backdoor, password cracking, and ransomware attacks. From each attack type, we selected three individual packets and converted them into human-readable JavaScript Object Notation (JSON) format using Wireshark’s default fields. Each packet was then used to prompt the LLM 12 times, with each prompt asking whether the packet belongs to a specific attack type. In total, we queried Llama3.1:70b 468 times with unique prompts. Finally, we manually analysed the results.

Figure 1. Research Process

During the analysis of Llama 3.1, we examined how often it categorises a packet as an attack. We also more closely examined the types of malicious activities it recognises, how rational its responses are, and its current limitations. The following five statements helped structure our analysis:

• True findings (TF)

• The findings of Llama 3.1 were from the provided data, such as correct Internet protocol (IP) addresses and port numbers.

• False findings (FF)

• Llama 3.1 hallucinated the findings so much that it affected the quality of the response, including additional communication layers and wrong MAC addresses.

• Attack vector (AV) identification

• Llama 3.1 identified signals of potentially malicious activities from the data and mentioned that the findings were suspicious. Llama 3.1 did not need to specify the exact attack type but should provide enough suspicious findings for a user to deduce ongoing malicious action.

• Correct reasoning (CR)

• Llama 3.1 drew correct conclusions based on its own findings, regardless of whether those findings were accurate or hallucinated. We evaluated only the correctness of the conclusions.

• False reasoning (FR)

• The conclusions made by Llama 3.1 based on the findings were wrong.

During the analysis, we focused on the findings and reasoning essential to the conclusions, disregarding minor mistakes. The key criterion was that the answer should be genuinely useful to a user. A single minor suspicious finding (e.g. an unusual port number) was insufficient for success. However, Llama 3.1 did not need to identify all suspicious elements a human might find; one major finding (e.g. an SQL injection) or several smaller ones were enough.

B. Prompt engineering

The prompt below shows the structure used throughout the Llama 3.1 analysis. Each prompt began with a single network packet to be analysed (see Section III-E for a more detailed description). After the packet, we added a brief context and then the actual task for the LLM. Each packet was prompted 12 times with different attack types. Prompts were independent so that previous prompts and responses were not available to the model.

Prompt 1: Prompt Structure Used in the Llama 3.1 analysis

At the end of the prompt, we added two prompt engineering methods to improve the performance of the LLM. In [32], the authors introduced the question-answer prompting (QAP) method, whose original form is ‘Explain this problem to me in at least n words. Then solve for the answer’. They used their method in arithmetic and commonsense problems, where the problem is described at the beginning of the prompt. Their problem description corresponds to the data in our case, so we changed the word ‘problem’ to ‘data’ in our prompt. We arrived at this solution because LLMs are sensitive to word choices, as also mentioned by [29]. This also allowed us to ensure whether the LLM had understood the provided data correctly. Reference [33] prompted LLMs to think step-by-step in their chain-of-thought method. We also adopted this method for our prompt.

C. Large language models

This study considered cyberattacks in IoT networks, which are used across various fields. For example, in health care, data privacy is a critical concern. In some networks, devices may also have only a very narrowband Internet connection, but an attack detection tool should still function effectively. These particularities of IoT networks led us to evaluate local LLMs in this study. We selected Llama as the model to be explored in depth because Llama models have performed well in various benchmark tests, and the most recent versions provide a relatively long context length of 128 K tokens [34]. At the starting point of this study, the latest Llama version was Llama 3.1. We chose the 70-billion-parameter version, Llama 3.1:70b, to strike a balance between improved performance and computational requirements.

As mentioned earlier, the heterogeneity of IoT network traffic data is one of the most significant issues related to IoT networks. Therefore, we chose not to fine-tune the tested model for a specific dataset. Instead, we tested how well LLMs could handle network traffic data using their general knowledge. Reduction of professional effort was also a key motivator of our study, and fine-tuning would have increased it.

We wrote our code using the Python programming language and utilised the Ollama framework [35] to run LLMs locally on our computer. The computer used has an Apple M3 Max chip with 128 GB of memory.

D. Data

To enable controlled testing of LLMs, we required a dataset consisting of cyber-related IoT network traffic that is labelled by attack type and exhibits heterogeneity in terms of, for example, devices and/or protocols. Edge-IIoTset [36] contains labelled network data generated from an IIoT network that includes more than 10 devices, using protocols such as IP, address resolution protocol (ARP), transmission control protocol (TCP), user datagram protocol (UDP), and Internet control message protocol (ICMP). The network has been attacked by 12 attack types, namely port scanning, vulnerability scanning attack, four different variations of DDoS attacks, OS fingerprinting, DNS spoofing attack, ARP spoofing attack, XSS attack, SQL injection, uploading attack, backdoor attack, password cracking attack, and ransomware attack. In addition, the dataset contains ten benign traffic files. The data are shared in CSV and PCAP formats.

An advanced persistent threat (APT) attack is commonly presented as a multiphase attack chain. Several models have been developed to describe APT attacks, such as the Cyber Kill Chain [37], MITRE ATT&CK [38], and Unified Kill Chain [39], which divide the attack chain into 7, 14, and 18 phases, respectively. These models describe the attack from different perspectives; thus, the content and scope of the phases vary. However, each model includes similar phases. For example, they begin with a reconnaissance phase, followed by resource development, initial access, installation of malicious code, command and control, and finally the accomplishment of the original goals. Techniques such as vulnerability and port scanning are used during the reconnaissance phase, while a backdoor attack is part of the code installation phase, typically launched before the actual malware, such as ransomware. Thus, the attacks in the Edge-IIoTset dataset represent only certain phases of an ATP attack. However, because the phases are named and separated as if they were distinct attacks in the dataset, this study also treated them more as separate attacks.

We used packets from PCAP files and decided to analyse individual packets as they are the basic unit of network traffic. We are aware that some attacks are challenging or even impossible to detect from a single packet. However, we aim to understand the capabilities of LLMs in analysing individual packets to later compare their performance with multi-packet approaches and consequently our future research will expand traffic analysis to include all packets within a TCP flow window. We randomly selected three packets from each attack type and benign traffic for our usage. In total, we used 39 individual packets in our analysis. A more detailed description of the data selection is presented in Appendix B.

We decided not to preprocess the data used. Our motivation behind this decision was the heterogeneity of IoT traffic data as one of our main research focus areas. We studied how well LLMs understand network traffic data without data preprocessing and evaluated the kind of preprocessing that may be essential to achieve good results. Usually, data professionals select the features, clean and standardise the data, and otherwise preprocess it before training ML models. This is an essential but time-consuming process when using traditional ML models. If LLMs were able to understand unprocessed data and detect cyberattacks from it, it would be a waste of time to preprocess the data.

E. Results

Table 1 summarises the results of the Llama 3.1 analysis. These results provide a foundation for the following findings.

Table 1. Summary of results of llama analysis

Our prompt requested Llama 3.1 to provide a probability of an attack for each packet. In all the experiments with malicious packets, Llama 3.1 labelled packet captures as a cyberattack in 56% of the cases (with a probability of 50% or higher provided by LLM) but could find and describe the signals of malicious activities in only 38% of the cases. This shows that the ability of Llama 3.1 to provide a correct answer does not necessarily mean that it eventually knows the answer or can justify it. This phenomenon was especially clear in our results during the ransomware and backdoor attacks.

According to our results, Llama 3.1 identified malicious activities best with SQL injection attacks. It achieved 86% and 92% accuracy in identifying the attack and describing malicious activities, respectively. This represents how Llama 3.1 successfully mentioned suspicious activities even if it was not sure about the attack. Similarly, Llama 3.1 detected SQL queries from the data and could specify that the packets contained an SQL injection attempt, even if it had been requested for other attack types. In addition to SQL injections, Llama 3.1 easily detected other language-based suspicious signals, such as XSS, password cracking, and uploading attacks. During the LLM comparison, we further investigated the ability of the LLMs to detect language-based malicious activities and categorise them correctly.

Llama 3.1 identified every packet as an attack at least once, but it could not detect suspicious activities for identifying specific attack type as often. In addition, it misclassified benign packets as malicious in 8 out of 36 prompts. The most challenging attack types for Llama 3.1 were ransomware attacks, OS fingerprinting, DNS spoofing, backdoor attacks, and ARP spoofing. The poor success in detecting OS fingerprinting, DNS spoofing, and ARP spoofing can be explained by the fact that these attacks are difficult to identify from a single packet, even for humans, as the traffic closely resembles normal activity.

The prompts where the requested attack type matched the data attack type had better results than the average of all prompts. Especially the success rate in detecting malicious activities increased remarkably from 38% to 58% when the prompt suggested a similar attack type to the actual attack type.

We evaluated the ability of Llama 3.1 to understand network traffic data by analysing its findings from the data. Llama 3.1 described the data with correct findings 87% of the time and added incorrect content to its response (hallucinated) 22% of the time. Our results show that Llama 3.1 had both true and hallucinated findings 9% of the time, which proved that the overall border between true and hallucinated responses is not always clear. We considered mainly the findings that affected the actual answer and did not care about minor mistakes.

As mentioned earlier, Llama 3.1 identified ransomware and backdoor attacks as malicious packets on average, but it struggled to justify these classifications. Even though providing justifications for such attacks can be challenging, the hallucination rate of Llama 3.1 was remarkably high in these cases. These cases involved encrypted payloads (ransomware packets 2–3 and backdoor packets 1–3). Other unreasonable data also seem to cause hallucination, such as a payload consisting entirely of 58 s (DDoS packet 3), a payload containing mainly 00 s (port scanning packet 2), and hexadecimal-formatted TCP options (XSS packets 2 and 3). However, Llama 3.1 can successfully read ASCII-formatted payloads (e.g. DDoS packet 2). In these cases, the hallucination rate was relatively low, even when the text was not human readable. These findings led us to examine how encrypted data affect LLM behaviour during the comparison.

The cybersecurity knowledge of Llama 3.1 was tested by analysing its reasoning on the basis of its own findings. We did not analyse the correctness of the final answers because the hallucinated findings would have distorted the results. Instead, we assessed whether the answer was correct and sensible from a cybersecurity perspective, using the findings of Llama 3.1 as a basis. For example, Llama 3.1 claimed that a packet was a SYN packet and used this information as evidence of a potential DDoS attack, even though in reality the packet was not a SYN packet. In this case, the claim about the SYN flag was a false finding but using it as part of the reasoning for a potential DDoS attack was still logically consistent.

The responses of Llama 3.1 contained correct reasoning 94% of the time. As success in correct reasoning was higher than success in correct findings (87%), we can say that Llama 3.1 can generate a believable answer almost always, even if it hallucinates. This demonstrates that in the absence of original data as a reference point, hallucinated findings and answers may be challenging to detect based on the response alone. Hallucinations and challenges in detecting hallucinated responses may also reduce the reliability of the model’s responses.

We found Llama 3.1 to make false reasoning 11% of the time, as seen in Table 1. This illustrates how often Llama 3.1 did not understand the cybersecurity relevance of its findings, thus indicating challenging packets. Although faulty reasoning usually occurs at the same time as hallucinations (e.g. ransomware attacks), this is not the case every time (e.g. ARP spoofing). The mistakes of Llama 3.1 were related to both the network traffic understanding and the knowledge of malicious activities. One example of a lack of understanding of network traffic was when Llama 3.1 recognised a packet as an ARP probe only once out of 12 prompts and failed to understand its purpose (ARP packet 2). More examples are described in the next section. However, our tests included several packets in which Llama 3.1 did not make a single false reasoning. These observations indicate that Llama 3.1 has some level of knowledge in the field of IoT network traffic and cybersecurity, but its understanding and reasoning ability have limitations, especially for rarer cases.

G. Strengths and weaknesses of Llama 3.1

Llama 3.1 successfully identified various signs of attacks from network traffic data. In this section, we present examples that demonstrate the ability of LLMs to understand network data and its context. These examples are presented in Table 2. We selected examples that illustrate how Llama 3.1 operated and those that occurred repeatedly. The goal of this section is to enhance the understanding of the strengths and weaknesses of Llama 3.1 as a cyberattack detector.

Table 2. Llama analysis for single packets

On the basis of our results, Llama 3.1 demonstrated a general understanding of how different protocols operate, their purposes, and the types of data they typically transfer. The meaning of parameters was usually correctly understood. Llama 3.1 was able to consider the context of an IIoT network (Example 2 in Table 2) and the features of the HTTP methods used (Example 1 in Table 2) in its reasoning. However, Llama 3.1 struggled with, for instance, less common ARP packets as described earlier.

The most common observations made by Llama 3.1 from packet captures were MAC addresses, IP addresses, and port numbers. Llama 3.1 has a good, though imperfect, understanding of the most common default ports for different protocols, such as HTTP using port 80 and HTTPS using port 443. It also considered high-numbered ports suspicious (examples 4–7 in Table 2). In addition, Llama 3.1 understands the structure of IPv4 addresses, can determine when devices are in the same local network, and uses this information to estimate the likelihood of a cyberattack (Example 3 in Table 2).

Llama 3.1 can identify human-readable suspicious pieces of data and make conclusions based on them. The model was reliable in detecting SQL queries, usually from URLs (Example 11 in Table 2) and often detected suspicious payloads where data seemed maliciously generated. For instance, Llama 3.1 identified if the payload contained mainly continuous character sequences, such as …00:00:00:00…, 58:58:58:58… or 08:09:0a:0b… (examples 8 and 9 in Table 2). At the same time, however, such nonsensical payloads seem to induce hallucinations.

Llama 3.1 can identify uploaded files as potentially malicious even without access to their actual content (examples 10 and 13 in Table 2I), describe suspicious purposes or vulnerabilities associated with various tools (examples 14–18 in Table 2), and read ASCII-formatted data (Example 19 in Table 2). This finding inspires us to investigate whether LLMs can interpret network traffic packets in hexadecimal format.

As mentioned previously, Llama 3.1 hallucinated frequently when its prompt included encrypted or other data that appeared nonsensical from its perspective. In our prompt, we requested Llama 3.1 to both describe and analyse the data. This approach provided us with a highly valuable view of the behaviour of LLMs in the case of hallucinations. Llama 3.1 not only added hallucinated values, such as additional layers, but also hallucinated existing parameters, such as IP addresses and port numbers. Example 20 in Table 2shows a response where the MAC address was extracted from the payload. The hallucinated IP and MAC addresses were also often strings that did not exist in the provided data, likely originating from the training data of the model. During the LLM comparison, we investigated this behaviour with various local LLMs.

In the case of DNS spoofing, Llama 3.1 did not find short TTL values (TTL  =  1) or the lack of DNS response as suspicious, even if the provided data contained a note or warning by Wireshark. Overall, Llama 3.1 had a high variation in identifying attacks across attack types and packets. Thus, it has promising capabilities but is not reliable enough to operate on its own.

On the basis of our in-depth analysis, Llama 3.1:70B has a general but limited understanding of the context of IoT network traffic data and can recognise attributes that should not appear in normal traffic. It is also familiar with various types of cyberattacks and can identify especially language-based indicators of malicious activity from network traffic. In addition, it can interpret ASCII-formatted data and demonstrates strong reasoning capabilities.

However, we observed that Llama 3.1 became confused when encrypted data are included in the prompt. In such cases, it only occasionally provided sensible responses. Furthermore, although Llama 3.1 could identify signs of attacks in the provided data, this ability was inconsistent. It often failed to detect clear signs that it successfully identified in other runs. Its knowledge also has limitations, especially in handling rarer cases.

Some of our experiments were difficult to identify as an attack based on only one packet, especially ARP spoofing. Thus, we provided more information for Llama 3.1 by adding two packets for our prompt, which together made the ARP spoofing detection possible. We used the same prompt as earlier, except that the text ‘The file contains a single packet’ was replaced with the correct number of packets. A part of the Llama 3.1 analysis is added in Table 3, demonstrating that Llama 3.1 can utilise multiple packets while analysing network traffic data. Using larger amounts of data would enable a wider analysis of network traffic.

Table 3. Llama 3.1 analysis using multiple packets

IV. LLM comparison

A. Research process

During our in-depth analysis of the ability of Llama 3.1 to interpret network traffic data, we identified several interesting findings. However, because the Llama 3.1 analysis focused only on Llama 3.1, the findings may not be applicable to other LLMs. This limitation led us to develop targeted experiments to examine the behaviour of various LLMs.

We developed eight experiments to compare several LLMs. First, we evaluated their ability to understand and describe network traffic data presented in various formats. The subsequent tests focused on how well LLMs can detect cyberattacks from individual traffic packets.

The goal of experiments 1–4 was to determine if LLMs understand hexadecimal-formatted network traffic data and if removing encrypted payloads reduces hallucination. We selected 200 packets from the Edge-IIoT dataset, including 100 from the ransomware attack and 100 from the backdoor attack. More details of the dataset selection are presented in Appendix B. We prompted each model with the same prompt four times per packet. In each run, we used the same packet but presented it in different formats: hexadecimal, hexadecimal without (encrypted) payload, human readable, and human-readable without (encrypted) payload.

In experiments 5–8, we examined whether LLMs can detect malicious activities from network traffic that are expressed through language-based patterns, such as SQL queries, and identify the types of attacks. The task of LLMs was to categorise the packet as malicious or benign and then identify the attack type. We selected 100 packets from four attack types, namely SQL injection, uploading, password cracking, and benign. This attack type selection kept the cases as simple as possible; that is, the payloads included suspicious parts in human-readable format, and benign traffic contained only TCP handshake packets. These attacks were also detectable from single packets. The dataset selection criteria are detailed in Appendix B.

B. Prompt engineering and LLMs

We selected five recent local LLMs with various sizes to be tested: Gemma 3:12B, Gemma 3:4B, Llama 3.1:70B, Llama 3.2:3B, and Phi-3 Medium (14B).

Prompt 2: LLM comparison (experiments 1-4)

During the first four experiments, we used the prompt above with various packet formats. The prompt contains similar elements to those in the Llama 3.1 analysis, but the focus was on finding the IP and MAC addresses and port numbers from the data. In addition, we included a structured format for the response to enable automatic response validation.

Prompt 3: LLM comparison (experiments 5-8)

During experiments 5–8, we examined a situation where another IDS detected a packet as a malicious packet. We used the same prompt with each experiment, changing only the attack type between the experiments. All the packets were presented in human-readable format.

C. Results

In experiments 1–4, our requirements for the response were that the LLMs must identify and mention the correct attributes (IP and MAC addresses and port numbers). In addition, the models were expected to follow the given format, for example, ‘Source IP:’; otherwise, we left the corresponding value empty.

Figure 2 outlines the results of experiments 1–4. None of the tested LLMs were able to understand network traffic data in hexadecimal format, even after the encrypted payload was removed. With a human-readable format, Gemma 3:12B and Llama 3.1:70B were almost perfectly able to identify the requested attributes. However, the other models struggled more, especially with MAC addresses.

Figure 2: Impact of hexadecimal representation and encrypted payload on the LLM analysis

In Experiment 3, smaller models (Gemma 3:4B, Llama 3.2:3B, and Phi-3 Medium) struggled to extract the requested information from human-readable packets. We observed that removing the encrypted payload from the packet improved the performance of LLMs across all attributes. This finding reinforces the notion that encrypted data adversely affects the performance of language models. However, in experiments 3 and 4, the larger models (Gemma 3:12B and Llama 3.1:70B) answered almost perfectly, even when the packet contained encrypted payloads. Therefore, encrypted data had only a minor impact on their performance, although Llama 3.1:70B struggled with encrypted data during the Llama 3.1 analysis. This underlines the importance of prompt engineering.

During Experiment 4, Gemma 3:4B consistently identified destination and source MAC addresses as the same. Overall, it appears that larger models performed better and smaller ones performed worse with encrypted data, although model size is not the only explanatory factor, as Phi-3 Medium has 14B parameters.

The results of the classifications as benign or malicious in experiments 5–8 are shown in Table 4. We used several evaluation metrics to enable fair model comparison: accuracy for single attack types and, for overall comparison, balanced accuracy (BA), precision, recall, and F1-score. Recall and precision were used as defined in [40], where the positive classification refers to benign traffic. In the case of Gemma 3:4B and Gemma3:12B, precision is undefined because it involves division by zero, as there were zero packets labelled as benign.

Table 4: Attack Identification (experiments 5-8)

Phi-3 Medium performed best according to most metrics, achieving an F1-score of 0.83 and a balanced accuracy of 0.91. Llama 3.1:70B, Gemma 3:4B, and Gemma 3:12B performed best in identifying malicious packets. However, they had the lowest success rates for benign packets, resulting in very low performance across the most metrics. In contrast, Llama 3.2:3B and Phi-3 Medium performed well with benign packets but struggled the most with classifying attack packets. Overall, the evaluated local LLMs varied significantly in performance and often overemphasised either the malicious or the benign class.

During experiments 5–8, the task of LLMs was also to identify the attack type. The following are the accepted attack types that the LLMs mentioned:

• SQL injection: ‘SQL’

• Uploading Attack: ‘uploading,’ ‘remote code execution,’ and ‘code injection’

• Password Cracking: ‘password,’ ‘credential,’ ‘brute force,’ and ‘brute-force’

• Benign: ‘benign,’ ‘n/a,’ ‘not applicable,’ ‘unknown,’ and ‘legitimate’

A comparison between correct attack identification and attack type classification is shown in Figure 3. Attack identified correctly refers to cases in which the LLM categorised packets accurately as either malicious or benign, whereas Attack type identified refers to cases in which the attack type was correctly labelled. As expected, the LLMs identified attacks more accurately than attack types. The easiest attack type to identify was SQL injection, where only Llama 3.2:3B struggled. Password cracking was also relatively easy for the LLMs to detect. By contrast, uploading attack and benign packets were much harder for the LLMs to identify correctly. In these cases, the LLMs often mentioned incorrectly vulnerability scanning, which was included as an example in the prompt.

FIgure 3: Attack identification and attack type categorization

Overall, the results of the LLM comparison confirmed the findings of the Llama 3.1 analysis. In addition, it provided current information on the strengths of the LLMs and differences between them. Hexadecimal-formatted data appear to be too complex for current local LLMs, and encrypted data weaken the performance of LLMs.

Our results show that the success of LLMs in understanding network traffic data and detecting malicious packets varies between models and attack types. Overall, LLMs have general knowledge of network traffic data and cyberattack detection, even without further learning. The largest models appeared to perform better; however, the large size of the model is not a guarantee of success. Moreover, one model seemed good at identifying malicious activities, whereas the other was better with benign packets. Thus, in network traffic analysis, the best approach may be a combination of several techniques where LLMs perform only in their strength areas, such as language-based tasks.

V. Discussion

On the basis of our analysis, Llama 3.1 has a general but limited knowledge of various cyberattack types and IoT network traffic. It offers many advantages compared with other models, such as traditional ML and DL models. Unlike traditional models, Llama 3.1 was able to understand the context of the provided data and demonstrated knowledge of cyberattacks, protocols, and tools, at least to some extent. In addition, it comprehended human-readable language and ASCII characters and made logical conclusions. Llama 3.1 could also explain what happened in the data and identify various potential signs of malicious activity. On the other hand, it is not capable of processing machine-readable data, such as hexadecimal-formatted packets.

In this study, we showed that it is possible to use LLMs without fine-tuning or context learning to analyse raw network traffic data. Our method allowed us to observe how LLMs work with unfamiliar datasets with various protocols. Our study showed that LLMs did not require network-specific information about normal traffic to analyse network traffic data. Thus, LLMs may be valuable tools when handling heterogeneous data because it does not need to define all the network characteristics. However, providing a small amount of information can still be useful, such as a list of trusted IP addresses. More research is needed to determine the necessary information.

The LLMs tested in this research are not yet reliable enough to analyse network traffic data independently. However, we see that larger models may be suitable for searching and explaining malicious activities from network traffic data, but human involvement is still needed. For example, LLMs could be part of an anomaly detection -based next-generation (NG) firewall that also includes other services, such as VPN, NAT, and routing. LLMs would operate at the packet level and focus on threat types for which they perform best, such as detecting SQL injections. They could be applied using in-depth inspection principles and function as a filtering component within the system. In addition, LLMs do not need to be updated or retrained as frequently as traditional ML models, which is another strength of LLMs in the prevailing fast changing threat landscape. Other modules could handle different types of malicious activity within their respective strengths, ensuring that resources are used as efficiently as possible. In such a setup, LLMs could be located in an NG firewall in the demilitarised zone (DMZ), where they could assist professionals in analysing external network data flagged as malicious by an IDS in DMZ. Thus, they could make the data analysis process quicker and more autonomous. It is still essential to use a human-readable format of network traffic data and ensure that encrypted data do not cause hallucination.

LLMs still have multiple weaknesses that need solutions. The general weaknesses are the computational requirements and their habit of hallucinating. The analysis of packets took a long time; thus, it is not meaningful to analyse real-time network traffic packets one by one using LLMs. In addition, the calculation requires a lot of memory and power, which can be limited in IoT networks. The use of LLMs also requires a mature interface.

VI. Limitations and future research

Our study provides an in-depth examination of Llama 3.1 and a comparison of five recent local LLMs. We used one dataset in our research. The dataset was comprehensive and included several attack types, but it did not contain real-life network traffic. The data structure was also partially confusing. For example, the port scanning attack contained only scanning traffic, without any benign traffic mixed with the malicious traffic. Moreover, backdoor installation is part of all ATP attacks, and searching them individually may distort the results, as the probability of various attack types does not correspond to reality. Therefore, teaching the models with this dataset may cause biases. However, we analysed individual packets, and thus the structural biases do not affect our research and achieved results. Since the content of attacking packets is similar regardless of the dataset, to the best of our knowledge, using additional datasets would not add value in a single packet data-based analysis. Verification of this is one future research topic. Forthcoming research will include more datasets, particularly benign and real-life IoT network traffic data from our internet connected research environment.

Approaches that combine multiple techniques and alternative prompting methods should also be explored for IoT network-related cybersecurity tasks, as it remains unclear, for example, how LLMs understand the term typical traffic in the prompt used during the Llama 3.1 analysis. LLMs are a rapidly evolving technology, and research such as the present study may be required regularly to identify model-specific strengths and weaknesses and to determine when models are advanced enough for reliable analysis.

Our results indicate that LLMs require data preprocessing (e.g. converting data into a human-readable format) and may benefit from more detailed prompts to better understand IoT network traffic data. It is important to investigate whether additional network information can enhance the performance of LLMs while maintaining their ability to generalise. Future research directions also include using larger amounts of data in a single prompt and examining the effects of model fine-tuning and context learning.

VII. Conclusions

To our best knowledge, this is the first study to qualitatively analyse LLMs as cyberattack detectors using packet-level IoT network traffic data. We prompted Llama 3.1 with combinations of single network packet captures and potential attack types. Our method enabled us to derive several valuable and practical insights. For instance, Llama 3.1 can detect language-based malicious signals (e.g. SQL queries) but struggles with encrypted data.

Our research also examined five recent local LLMs and found that the behaviour observed in Llama 3.1 is largely consistent across the other models. Evaluating LLM performance requires more than just measuring success rate, as correct answers do not necessarily indicate that the model can justify its responses. Furthermore, LLMs can identify some cyberattacks from IoT network traffic with unusually little data preprocessing. These observations should be taken into account in future research and real-world applications.

Acknowledgements

This work was supported by the Austrian Research Promotion Agency (FFG), Business Finland (BF), the Centre for the Development of Industrial Technology (CDTI), the Swedish Agency for Innovation Systems (Vinnova) within the EUREKA CELTIC-NEXT project CISSAN under Grant 8365/31/2022, and the University of Jyväskylä.

Disclosure statement

No potential competing interest was reported by the authors.

Data availability statement

The data that support the findings of this study are openly available in IEEE DataPort at https://dx.doi.org/10.21227/mbc1-1h68, reference number [41].

Funding

This study was supported by Business Finland (8365/31/2022) and the University of Jyväskylä.

Notes on contributors

Sara-Päivi Paukkeri

Sara-Päivi Paukkeri received an MSc (Tech) degree in Information and Knowledge Management from Tampere University, Finland, in 2022. She is currently pursuing a PhD degree with the University of Jyväskylä. Her current research interests include cybersecurity, data-analysis, and LLMs.

Tapio Frantti

Tapio Frantti holds degrees of MSc, LicTech and Dr. Tech. from the University of Oulu. He is also an associate professor in the University of Oulu. He has worked at the Outokumpu Polarit Oy, University of Oulu, Nokia Telecommunications, and Nokia Mobile Phones as a researcher, senior researcher, chief engineer, and research manager. He has also worked in the Technical Research Centre of Finland as a chief research scientist and research professor and as a visiting professor in Tokyo Denki University. Lately, he has worked in Renesas Mobile Europe and Broadcom Communications Finland as a distinguished researcher and research leader, in IoLiving Ltd. as a CTO and as a co-director of Security and Software Engineering Research Centre. Currently, he works as a cybersecurity professor in the University of Jyväskylä. He has been in the field for around 30 years and he has published more than 100 scientific and technical papers in journals, magazines, books, and international conferences. He has also authored several patents. Tapio Frantti also acts as a regular reviewer in international conferences and 12 scientific top-rated journals. He is a member of technical programme committees in tens of international conferences. His research interest are in the adaptive and intelligent control theory, networking technologies, and cybersecurity.

References

[1] Statista (2024) Number of Internet of Things (IoT) connected devices worldwide from 2019 to 2033, by vertical,”
May. [Online]. Available: https://www.statista.com/statistics/1194682/iot-connected-devices-vertically/.

[2] Jafari M., Majidi F., Heydarnoori A. Prioritizing app reviews for developer responses on google play. arXiv:
2502.01520v1. 2025;10.18293/DMSVIVA2024-153

[3] Kolasa K., Admassu B., Hołownia-Voloskova M., et al. Systematic reviews of machine learning in healthcare: a
literature review. Expert Rev Pharmacoeconomics Outcomes Res. 2024;24(no. 1):63–115. doi: https://doi.org/
10.1080/14737167.2023.2279107

[4] Nazir R., Laghari A. A., Dahri F. H., et al. A review on machine learning techniques for network security. J Cyber
Secur Technol. 2026;10(1):1–45. doi: https://doi.org/10.1080/23742917.2025.2480730

[5] IBM (2024) “Cost of a Data Breach Report 2024”. [Online]. Available: https://www.ibm.com/reports/data-breach

[6] Check Point Research (2023) “The Tipping Point: Exploring the Surge in IoT Cyberattacks Globally,” [Online]. Available:
https://blog.checkpoint.com/security/the-tipping-point-exploring-the-surge-in-iot-cyberattacks-plaguing-the-educa-
tion-sector/

[7] Gupta M., Akiri C., Aryal K., et al. From ChatGPT to ThreatGPT: impact of generative AI in cybersecurity and privacy.
IEEE Access. 2023;11:80218–80245. doi: 10.1109/ACCESS.2023.3300381

[8] Al-Hawawreh M., Aljuhani A., Jararweh Y. Chatgpt for cybersecurity: practical applications, challenges, and future
directions. Clust Comput. 2023;26(6):3421–3436. doi: 10.1007/s10586-023-04124-5

[9] Alawida M., Abu Shawar B., Abiodun O. I., et al. Unveiling the dark side of ChatGPT: exploring cyberattacks and
enhancing user awareness. Information (Basel). 2024;15(1):27. doi: 10.3390/info15010027

[10] Fang R., Bindu R., Gupta A. LLM agents can autonomously hack websites. arXiv:2402.06664v3. 2024 Feb.

[11] Happe A., Cito J. Getting pwn’d by AI: penetration testing with large language models In: ESEC/FSE ’23. San
Francisco, CA, USA: ACM; Dec. 3-9, 2023. pp. 2082–2086 10.1145/3611643.3613083

[12] Ferrag M. A., Battah A., Tihanyi N. SecureFalcon: the next cyber reasoning system for cyber security. arXiv:
2307.06616v1. 2023 Jul.

[13] Ragsdale J., Boppana R. V. On designing low-risk honeypots using generative pre-trained transformer models
with curated inputs. IEEE Access. 2023;11:117528–117545. doi: 10.1109/ACCESS.2023.3326104

[14] McKee F., Noever D. Chatbots in a honeypot world. arXiv:2301.03771v1. 2023 Jan .

[15] Sladić M., Valeros V., Catania C. LLM in the shell: generative honeypots In: EuroS&P. Vienna, Austria: IEEE; Jul. 8-12,
2024. pp. 430–435 10.1109/EUROSPW61312.2024.00054

[16] Jin H, Papadimitriou G, Raghavan K. Large language models for anomaly detection in computational workflows:
from supervised fine-tuning to in-context learning In: SC24. Atlanta, GA, USA: IEEE; Nov. 17-22, 2024. pp. 1–17
10.1109/SC41406.2024.00098

[17] Makhdoomi P. M. S., Ikhlas M., Khursheed A., et al. Network-based intrusion detection: a comparative analysis of
machine learning approaches for improved security. J Cyber Secur Technol. 2026;10(1):1–28. doi: https://doi.org/
10.1080/23742917.2024.2447119

[18] Saif S., Ansari A. A., Biswas S., et al. A comprehensive analysis of machine learning-based intrusion detection
systems: evaluating datasets and algorithms for Internet of things. J Cyber Secur Technol. 2026;10(1):1–27. doi:
https://doi.org/10.1080/23742917.2024.2447124

[19] Houssel P. R. B., Singh P., Layeghy S., et al. Towards explainable network intrusion detection using large language
models. arXiv:2408.04342v1. 2024 Aug .

[20] Arif Iftakher Mahmood M., Ashab F., Saifuzzaman Sohan M., et al. LLM-Enhanced security framework for IoT
network: anomaly detection and malicious devices identification. IEEE Access. 2025;13:168405–168419. doi:
10.1109/ACCESS.2025.3613588

[21] S. Baral, S. Saha and A. Haque, “An adaptive end-to-end IoT security framework using explainable AI and LLMs,” in
2024 IEEE 10th World Forum on Internet of Things (WF-IoT), Ottawa, ON, Canada, 2024, pp. 469-474, 10.1109/WF-
IoT62078.2024.10811456.

[22] Diaf A., Korba A. A., Karabadji N. E., et al. Beyond detection: leveraging large language models for cyber attack
prediction in IoT networks In: 20th Int. Conf. DCOSS-IoT. Abu Dhabi, United Arab EmiratesApril 29–May 1, 2024.
pp. 117–123ISBN: 979-8-3503-6944-1, 10.1109/DCOSS-IoT61029.2024.00026

[23] Jerabek K., Koumar J., Setinský J., et al. Explainable anomaly detection in network traffic using LLM In: NOMS
2025-2025 IEEE Network Operations and Management Symposium. Honolulu, HI, USA2025. pp. 1–6 10.1109/
NOMS57970.2025.11073574

[24] Ferrag M. A., Ndhlovu M., Tihanyi N., et al. Revolutionizing cyber threat detection with large language models: a privacy-
preserving BERT-based lightweight model for IoT/IIoT devices. IEEE Access. 2024;1210.1109/ACCESS.2024.3363469

[25] F. Adjewa, M. Esseghir, L. Merghem-Boulahia and C. Kacfah, “LLM-based continuous intrusion detection frame-
work for next-gen networks,” 2025 International Wireless Communications and Mobile Computing (IWCMC), Abu
Dhabi, United Arab Emirates, 2025, pp. 1198-1203, 10.1109/IWCMC65282.2025.11059643.

[26] Zhang H., Bin Sediq A., Afana A., et al. Large language models in wireless application design: in-context learning-
enhanced automatic network intrusion detection In: GLOBECOM 2024 - 2024 IEEE Global Communications
Conference. Cape Town, South Africa2024. pp. 2479–2484 10.1109/GLOBECOM52923.2024.10901312
16 S.-P. PAUKKERI AND T. FRANTTI

[27] Arisdakessian S., Wahab O. A., Mourad A., et al. A survey on IoT intrusion detection: federated learning, game
theory, social psychology, and explainable AI as future directions. IEEE Internet Things J. 2023;10(5):4059–4092.
doi: 10.1109/JIOT.2022.3203249

[28] Xiong Z., Li W., Cai Z. Federated generative model on multi-source heterogeneous data in IoT. in AAAI-23.
2023;37(9):10537–10545. doi: 10.1609/aaai.v37i9.26252

[29] P. V. Dinh, D. T. Hoang, N. Q. Uy, D. N. Nguyen, S. P. Bao, and E. Dutkiewicz, “Multiple-input auto-encoder for IoT
intrusion detection systems with heterogeneous data,” ICC 2024, Denver, CO, USA, 2024, pp. 2707-2712, 10.1109/
ICC51166.2024.10622942.

[30] Chen D., Zhang F., Zhang X. Heterogeneous IoT intrusion detection based on fusion word embedding deep
transfer learning. IEEE Trans. Industr. Inform. 2023;19(8):1–10. doi: 10.1109/TII.2022.3227640

[31] Kumar N., Singh J. P., Kumar P. Machine learning-enhanced IoT network security: a black hole algorithm-based
feature selection approach for intrusion detection. J Cyber Secur Technol. 2026;10(1):1–19. doi: 10.1080/
23742917.2025.2542995

[32] Yugeswardeenoo D, Zhu K, O'Brien S. Question-analysis prompting improves LLM performance in reasoning
tasks. arXiv:2407.03624v2. 2024 Aug .

[33] Wei J., Wang X., Schuurmans D. Chain-of-thought prompting elicits reasoning in large language models.
arXiv:2201.11903v6. 2023 Jan .

[34] Llama Team and AI @ Meta (2024) The Llama 3 Herd of Models, arXiv:2407.21783v1

[35] Ollama Available: https://ollama.com

[36] Ferrag M. A., Friha O., Hamouda D., et al. Edge-IIoTset: a new comprehensive realistic cyber security dataset of IoT
and IIoT applications for centralized and federated learning. IEEE Access. 2022;10:40281–40306. doi: 10.1109/
ACCESS.2022.3165809

[37] LM ( n.d.) “Kyber Kill Chain,”, Lockheed Martin Corporation, Bethesda, MD, USA. Accessed: Nov. 4, 2025 [Online].
Available: https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill-chain.html

[38] MITRE ( n.d.) “MITRE ATT&CK,” The MIRTE Corporation, McLean, VA, USA. Accessed: Nov. 11, 2025. Available:
https://attack.mitre.org

[39] P. Pols. “The Unified Kill Chain,” 2023, Available: https://www.unifiedkillchain.com/assets/The-Unified-Kill-
Chain.pdf

[40] Danso P. K., Neto E. C. P., Dadkhah S. Ensemble-based intrusion detection for Internet of things devices In: HONET.
Marietta, GA, USA: IEEE; Dec. 19-21, 2022. pp. 034–03910.1109/HONET56683.2022.10019140

[41] Ferrag M. A., Friha O., Hamouda D. Edge-IIoTset: a new comprehensive realistic cyber security dataset of IoT and
IIoT applications: centralized and federated learning. IEEE Dataport. 2022 Jan;10.21227/mbc1-1h68

Appendixes

Appendix A presents all the results of the Llama 3.1 analysis. Each prompt was evaluated on the basis of the five statements described in Section III A. In addition, we examined whether Llama 3.1 identified packets as an attack. The prompts, where the requested attack type matches the data attack type, are highlighted grey, and the corresponding calculated values have a light grey background. The ARP spoofing attack and benign traffic packets are marked with a hyphen in the AV identification column to indicate no clear signs of attacks (Figure 4).

In Appendix B, the detailed data selection during both the Llama 3.1 analysis and the LLM comparison is described.

Appendix A: Results of the Llama 3.1 Analysis

FIgure 4: Each prompt is rated based on attack identification (A) and five assertions: true findings (TF), false findings (FF), attack vector (AV) identification, correct reasoning (CR), and false reasoning (FR). True statements are indicated with a cross.

Appendix B: Dataset Selection

During the entire research process, we used the Edge-IIoTset dataset. The data used consisted of raw network traffic data in .pcap files.

Llama 3.1 Analysis

For the Llama 3.1 analysis, we randomly selected three individual packets from each attack type, namely port scanning, vulnerability scanning, four different types of DDoS attacks, OS fingerprinting, DNS spoofing, ARP spoofing, XSS, SQL injection, uploading, backdoor, password cracking, and ransomware. For the DDoS attacks, we selected three packets in total, one each from TCP SYN Flood attacks, UDP flood DDoS attacks, and HTTP flood DDoS attacks.

After selecting the packets, we manually checked whether they contained clues indicating the attack. If not, we repeated the selection process. ARP spoofing attacks were particularly challenging to detect from a single packet, so these packets did not contain clear attack indicators. The difficulty of detecting an attack varies significantly between the packets, ranging from suspicious port numbers to malicious SQL queries.

In addition, we selected three packets of normal traffic, one from each of the following .pcap files: Distance, Modbus, and Temperature and Humidity. After selection, we converted the raw .pcap captures to JSON format, including all default information added by Wireshark. No further preprocessing was performed.

LLM comparison

In experiments 1–4, we selected two sets of 100 packets each from the ransomware and backdoor attack files. We chose the first 100 packets from each file that contained encrypted payloads. Both attack types primarily consisted of traffic from the target device to the offensive device. These packets were formatted in various ways depending on the experiment:

Experiment 1: Packets in hexadecimal format

d8f2ca8e1769dca632fb69b50800450005dccdd440004006e4ccc0a80080c0a800aadc0210e199adc3bc569fb2c6801005b83e9a00000101080a2aeef05f3d58fc87376ac4ecfdac8ce083b38eaab730472c2e64d998376ac4ed376ac2d4376ac4ecd4d0307f54d8bbbd15f9e81eb01a5219f2601614a6961a14a3777d4226fcdc2607cca39aef2df1cb00d1f2d16a446a4cdbf0c818a4319eb29a08b547a05d2223b8ef1a7ef0a5f6c287059af9e1fac41f33dca2d694f0c6d49c5dd1f0c8641de676df850a55789bad36aeb423d1a8ae2d7355dd9e484b093f79d2000aa75f8d2ba7…

Experiment 2: Packets in hexadecimal format without payload

d8f2ca8e1769dca632fb69b50800450005dccdd440004006e4ccc0a80080c0a800aadc0210e199adc3bc569fb2c6801005b83e9a00000101080a2aeef05f3d58fc87

Experiment 3: Packets in human-readable format

<bound method Packet.show2 of <Ether dst=d8:f2:ca:8e:17:69 src=dc:a6:32:fb:69:b5 type=IPv4 |<IP version=4 ihl=5 tos=0x0 len=1500 id=52692 flags=DF frag=0 ttl=64 proto=tcp chksum=0xe4cc src=192.168.0.128 dst=192.168.0.170 |<TCP sport=56322 dport=rwhois seq=2578301884 ack=1453306566 dataofs=8 reserved=0 flags=A window=1464 chksum=0x3e9a urgptr=0 options=[('NOP', None), ('NOP', None), ('Timestamp', (720302175, 1029241991))] |<Raw load='7j\\xc4\\xec\\xfd\\xac\\x8c\\xe0\\x83\\xb3\\x8e\\xaa\\xb70G,.d٘7j\\xc4\\xed7j\\xc2\\xd47j\\xc4\\xec\\xd4\\xd00\x7fTػ\\xbd\x15\\xf9\\xe8\x1e\\xb0\x1aR\x19\\xf2`\x16\x14\\xa6\\x96\x1a\x14\\xa3w}B&\\xfc\\xdc&\x07̣\\x9a\\xef-\\xf1\\xcb\x00\\xd1\\xf2\\xd1jDjL\\xdb\\xf0\\xc8\x18\\xa41\\x9e\\xb2\\x9a\x08\\xb5G\\xa0]"#\\xb8\\xef\x1a~\\xf0\\xa5\\xf6\x87\x05\\x9a\\xf9\\xe1\\xfa\\xc4\x1f3ܢ֔\\xf0\\xc6Ԝ]\\xd1\\xf0\\xc8d\x1d\\xe6v߅\nUx\\x9b\\xad6\\xae\\xb4#Ѩ\\xae-sUݞHK\t?y\\xd2\x00\n\\xa7_\\x8d+\\xa7...' |>>>>>

Experiment 4: Packets in human-readable format without payload

  <  bound method Packet.show2 of  <  Ether dst  =  d8:f2:ca:8e:17:69 src  =  dc:a6:32:fb:69:b5 type  =  IPv4 |<IP version  =  4 ihl  =  5 tos  =  0x0 len  =  1500 id  =  52692 flags  =  DF frag  =  0 ttl  =  64 proto  =  tcp chksum  =  0xe4cc src  =  192.168.0.128 dst  =  192.168.0.170 |<TCP sport  =  56322 dport  =  rwhois seq  =  2578301884 ack  =  1453306566 dataofs  =  8 reserved  =  0 flags  =  A window  =  1464 chksum  =  0x3e9a urgptr  =  0 options  =  [('NOP', None), ('NOP', None), ('Timestamp', (720302175, 1029241991))] |» » 

For experiments 5–8, we selected attack types that included language-based signs of malicious activity. Specifically, we chose SQL injection, uploading, and password cracking attacks. Each attack type was analysed in a separate experiment, each involving 100 packets. In addition, we selected 100 benign packets. A more detailed description of the selection process is as follows:

Experiment 5: SQL injection

We selected the first 100 packets containing SQL queries. The IP packet length was greater than 100, and the source IP address was 192.168.0.170 (offensive device).

Experiment 6: Uploading attack

The uploading attack dataset included packets with a source IP address of 192.168.0.170 and an IP length greater than 200. We also verified that each packet contained an unencrypted payload.

Experiment 7: Password cracking

Each packet in the password cracking dataset contained login credentials. We selected packets where the source IP address was 192.168.0.170 and the IP length exceeded 300.

Experiment 8: Benign packets

Each benign packet was part of a TCP three-way handshake (connection establishment) or a TCP four-way handshake (connection termination). We selected 100 packets from the SQL injection .pcap file, where the IP flags were one of the following: S, SA, and FA.

To Cite This Article

Paukkeri, S. P., & Frantti, T. (2026). Evaluation of local LLMs for network traffic packet analysis. Journal of Cyber Security Technology, 10(1). https://doi.org/10.1080/23742917.2026.2671738

Licensing

© 2026 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The terms on which this article has been published allow the posting of the Accepted Manuscript in a repository by the author(s) or with their consent.

Disclaimer

The article published on this page was originally produced by and remains the intellectual property of its respective authors and the journal in which it was first published. It is reproduced here solely for informational purposes under the applicable terms under which it was made available.

Ioxa Research is not the author, publisher, or originator of this article and makes no claim of ownership over its contents. The views, opinions, findings, conclusions, and recommendations expressed in this article are those of the author(s) alone and do not represent the views or positions of Ioxa Research, its directors, officers, employees, or affiliates.

Ioxa Research expressly disclaims all liability for any loss, damage, or adverse consequences — whether direct, indirect, incidental, or consequential — arising from or in connection with any reliance placed upon the contents of this article by any person or entity.

This article is provided on an "as is" basis without warranty of any kind, express or implied.