Security Posts
Infocon: green
ISC Stormcast For Tuesday, March 19th, 2024 https://isc.sans.edu/podcastdetail/8900
Categorías: Security Posts
Ataque de Side-Channel a conversaciones con ChatGPT, CoPilot y otros LLMs
Hoy os vengo a hablar de un paper que fue publicado ayer mismo, y que me ha gustado mucho. Es una investigación que intenta descifrar la conversación que un usuario está teniendo con un LLM por medio de capturar la estructura de Tokens que se ha intercambiado, como side-channel.
Figura 1: Ataque de Side-Channel a conversaciones con ChatGPT, CoPilot y otros LLMs
Esto podría ser un problema clásico de criptografía y cripoanalisis, pero la gracia es que utilizan un LLM para que ayude a resolver el problema, os lo explico un poco para ver si queda más claro.
Figura 2: Libro de Cifrado de las comunicaciones digitales:
de la cifra clásica a RSA 2ª Edición de 0xWord
La idea es que tenemos un servicio que está utilizando un LLM, y éste servicio puede ser un chat, como ChatGPT o un CoPilot como el de Microsoft, donde el usuario está teniendo una conversación con el el motor LLM, que puede ser GPT3.5, GPT4, Llama2, o cualquier otro.
Figura 3: Secuencias Tokenizadas por GPT3.5 y GPT4
Cuando el modelo LLM envía la información protegida contra inspección, el Tokenizador envía los datos en Stream, es decir, que hay una secuencia continua de Tokens que deben ser marcados periódicamente para saber cuándo comienza y termina un Token para poder ser descifrado y enseñado al usuario, tal y como se ve en estos dos ejemplos de la Figura 3 de cómo los Tokenizadores de GPT3.5 y GPT4 funcionan.
Figura 4: What Was Your Prompt? A Remote Keylogging Attack on AI Assistants
Supongamos que el atacante es capaz de acceder a la secuencia cifrada enviada por el Tokenizador, ¿Podrá saber de qué se está hablando o qué está diciendo? Esta es la pregunta que trata de resolver el paper de "What Was Your Prompt? A Remote Keylogging Attack on AI Assistants", y que como podéis imaginar, la respuesta es sí para muchos entornos.
Figura 5: Servicios vulnerables a este side-channel para descifrar de qué se está hablando.
Y la respuesta es la que véis en la imagen anterior, donde la mayoría de los servicios son vulnerables a esta técnica de ataque. Y para hacerlo sólo hay que utilizar la tecnología que tenemos a nuestro alcance. Al final, existen tres técnicas que pueden ayudar a inferir de que se está hablando, que se van a usar en el modelo descrito en el gráfico anterior, y que son las siguientes:
1.- Utilizar un LLM entrenado para analizar Tokens y saber de qué puede ir la conversación: Como os podéis imaginar, es un problema donde un con un modelo entrenado con datos de cadenas de Tokens y tipos de conversaciones que se están teniendo, la inteligencia artificial generativa puede ayudar. Así, los investigadores han entrenado un LLM con estas conversaciones Tokenizadas para poder evaluar de qué se está hablando y qué se está diciendo en concreto.
2.- Utilizar el contexto para incrementar el ratio de acierto: Al final, las conversaciones con un LLM son cadenas tokenizadas secuenciadas, lo que hace que si se averigua una conversación en un secuencia, se le puede decir al LLM que averigüe el tipo de conversación en la secuencia+1 sabiendo que en la secuencia anterior se hablaba de un determinado tema. Esto incrementa el índice de aciertos. Lo mismo hacia atrás una vez que se ha terminado el proceso.
3.- Texto plano conocido: No hace falta decir que es posible saber qué determinadas respuestas se están diciendo de forma repetida. Es decir, ChatGPT utiliza más o menos las mismas respuestas para terminar sus conversaciones de determinados temas, o para declinar contestar porque el Prompt solicitado está dentro de los marcados como Harmfull Mode. Sabiendo estos textos en conversaciones, es más fácil entrena al modelo a reconocerlos y detectar las secuencias de Tokens que tienen que ver con ellos.
Figura 6: Modelo de evaluación de cadenas tokenizadas con un LLM
Con estos datos, los investigadores diseñaron el proceso que podéis ver en la Figura 6 continuación donde capturan las cadenas de tokens de las secuencias, y se las entregan al LLM para que pueda decir de qué se está hablando en una determinada conversación.
Figura 7: Ratio de éxito por tipo de conversación
Los resultados son sorprendentes, porque de media, en el 55% de los casos - como se ve en la Figura 7 - el modelo infiere correctamente el tema de conversación que se está teniendo entre el usuario y el modelo LLM, y en el 29% de media - como se ve en la imagen siguiente - de los casos fue posible reconstruir la estructura de conversación entre el modelo y el usuario, lo que permite acceder a mucha información.
Figura 9: Evaluación de rendimiento en evaluación de la estructura de conversación
Este sí que es un ataque que podría añadirse a la lista de problemas de seguridad por las que un servicio o una app que usa un LLM que tú hayas construido se pueda ver afectado, que no está aún en el OWASP TOP 10 de LLM Apps & Services, pero que deberás tenerlo presente.
Figura 10: Hacker & Developer in the Age of GenAI LLM Apps & Services
Esta es una técnica novedosa, que seguro que se puede aplicar no a conversaciones de LLMs con usuarios, sino con muchos otros modelos de comunicación en stream como side-channel, donde un modelo LLM entrenado con estas cadenas tokenizadas podría llegar a ser capaz de inferir parcial o totalmente la información transferida. Una pasada.
¡Saludos Malignos!
Autor: Chema Alonso (Contactar con Chema Alonso)
Sigue Un informático en el lado del mal RSS 0xWord
- Contacta con Chema Alonso en MyPublicInbox.com
Figura 1: Ataque de Side-Channel a conversaciones con ChatGPT, CoPilot y otros LLMs
Esto podría ser un problema clásico de criptografía y cripoanalisis, pero la gracia es que utilizan un LLM para que ayude a resolver el problema, os lo explico un poco para ver si queda más claro.
Figura 2: Libro de Cifrado de las comunicaciones digitales:
de la cifra clásica a RSA 2ª Edición de 0xWord
La idea es que tenemos un servicio que está utilizando un LLM, y éste servicio puede ser un chat, como ChatGPT o un CoPilot como el de Microsoft, donde el usuario está teniendo una conversación con el el motor LLM, que puede ser GPT3.5, GPT4, Llama2, o cualquier otro.
Figura 3: Secuencias Tokenizadas por GPT3.5 y GPT4
Cuando el modelo LLM envía la información protegida contra inspección, el Tokenizador envía los datos en Stream, es decir, que hay una secuencia continua de Tokens que deben ser marcados periódicamente para saber cuándo comienza y termina un Token para poder ser descifrado y enseñado al usuario, tal y como se ve en estos dos ejemplos de la Figura 3 de cómo los Tokenizadores de GPT3.5 y GPT4 funcionan.
Figura 4: What Was Your Prompt? A Remote Keylogging Attack on AI Assistants
Supongamos que el atacante es capaz de acceder a la secuencia cifrada enviada por el Tokenizador, ¿Podrá saber de qué se está hablando o qué está diciendo? Esta es la pregunta que trata de resolver el paper de "What Was Your Prompt? A Remote Keylogging Attack on AI Assistants", y que como podéis imaginar, la respuesta es sí para muchos entornos.
Figura 5: Servicios vulnerables a este side-channel para descifrar de qué se está hablando.
Y la respuesta es la que véis en la imagen anterior, donde la mayoría de los servicios son vulnerables a esta técnica de ataque. Y para hacerlo sólo hay que utilizar la tecnología que tenemos a nuestro alcance. Al final, existen tres técnicas que pueden ayudar a inferir de que se está hablando, que se van a usar en el modelo descrito en el gráfico anterior, y que son las siguientes:
1.- Utilizar un LLM entrenado para analizar Tokens y saber de qué puede ir la conversación: Como os podéis imaginar, es un problema donde un con un modelo entrenado con datos de cadenas de Tokens y tipos de conversaciones que se están teniendo, la inteligencia artificial generativa puede ayudar. Así, los investigadores han entrenado un LLM con estas conversaciones Tokenizadas para poder evaluar de qué se está hablando y qué se está diciendo en concreto.
2.- Utilizar el contexto para incrementar el ratio de acierto: Al final, las conversaciones con un LLM son cadenas tokenizadas secuenciadas, lo que hace que si se averigua una conversación en un secuencia, se le puede decir al LLM que averigüe el tipo de conversación en la secuencia+1 sabiendo que en la secuencia anterior se hablaba de un determinado tema. Esto incrementa el índice de aciertos. Lo mismo hacia atrás una vez que se ha terminado el proceso.
3.- Texto plano conocido: No hace falta decir que es posible saber qué determinadas respuestas se están diciendo de forma repetida. Es decir, ChatGPT utiliza más o menos las mismas respuestas para terminar sus conversaciones de determinados temas, o para declinar contestar porque el Prompt solicitado está dentro de los marcados como Harmfull Mode. Sabiendo estos textos en conversaciones, es más fácil entrena al modelo a reconocerlos y detectar las secuencias de Tokens que tienen que ver con ellos.
Figura 6: Modelo de evaluación de cadenas tokenizadas con un LLM
Con estos datos, los investigadores diseñaron el proceso que podéis ver en la Figura 6 continuación donde capturan las cadenas de tokens de las secuencias, y se las entregan al LLM para que pueda decir de qué se está hablando en una determinada conversación.
Figura 7: Ratio de éxito por tipo de conversación
Los resultados son sorprendentes, porque de media, en el 55% de los casos - como se ve en la Figura 7 - el modelo infiere correctamente el tema de conversación que se está teniendo entre el usuario y el modelo LLM, y en el 29% de media - como se ve en la imagen siguiente - de los casos fue posible reconstruir la estructura de conversación entre el modelo y el usuario, lo que permite acceder a mucha información.
Figura 9: Evaluación de rendimiento en evaluación de la estructura de conversación
Este sí que es un ataque que podría añadirse a la lista de problemas de seguridad por las que un servicio o una app que usa un LLM que tú hayas construido se pueda ver afectado, que no está aún en el OWASP TOP 10 de LLM Apps & Services, pero que deberás tenerlo presente.
Figura 10: Hacker & Developer in the Age of GenAI LLM Apps & Services
Esta es una técnica novedosa, que seguro que se puede aplicar no a conversaciones de LLMs con usuarios, sino con muchos otros modelos de comunicación en stream como side-channel, donde un modelo LLM entrenado con estas cadenas tokenizadas podría llegar a ser capaz de inferir parcial o totalmente la información transferida. Una pasada.
¡Saludos Malignos!
Autor: Chema Alonso (Contactar con Chema Alonso)
Sigue Un informático en el lado del mal RSS 0xWord
- Contacta con Chema Alonso en MyPublicInbox.com
Categorías: Security Posts
ISC Stormcast For Tuesday, March 19th, 2024 https://isc.sans.edu/podcastdetail/8900, (Tue, Mar 19th)
(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.
Categorías: Security Posts
Apple may hire Google to power new iPhone AI features using Gemini—report
Enlarge (credit: Benj Edwards)
On Monday, Bloomberg reported that Apple is in talks to license Google's Gemini model to power AI features like Siri in a future iPhone software update coming later in 2024, according to people familiar with the situation. Apple has also reportedly conducted similar talks with ChatGPT maker OpenAI.
The potential integration of Google Gemini into iOS 18 could bring a range of new cloud-based (off-device) AI-powered features to Apple's smartphone, including image creation or essay writing based on simple prompts. However, the terms and branding of the agreement have not yet been finalized, and the implementation details remain unclear. The companies are unlikely to announce any deal until Apple's annual Worldwide Developers Conference in June.
Gemini could also bring new capabilities to Apple's widely criticized voice assistant, Siri, which trails newer AI assistants powered by large language models (LLMs) in understanding and responding to complex questions. Rumors of Apple's own internal frustration with Siri—and potential remedies—have been kicking around for some time. In January, 9to5Mac revealed that Apple had been conducting tests with a beta version of iOS 17.4 that used OpenAI's ChatGPT API to power Siri.Read 5 remaining paragraphs | Comments
Categorías: Security Posts
Fujitsu says it found malware on its corporate network, warns of possible data breach
Enlarge (credit: Getty Images)
Japan-based IT behemoth Fujitsu said it has discovered malware on its corporate network that may have allowed the people responsible to steal personal information from customers or other parties.
“We confirmed the presence of malware on several of our company's work computers, and as a result of an internal investigation, it was discovered that files containing personal information and customer information could be illegally taken out,” company officials wrote in a March 15 notification that went largely unnoticed until Monday. The company said it continued to “investigate the circumstances surrounding the malware's intrusion and whether information has been leaked.” There was no indication how many records were exposed or how many people may be affected.
Fujitsu employs 124,000 people worldwide and reported about $25 billion of revenue in its fiscal 2023, which ended at the end of last March. The company operates in 100 countries. Past customers include the Japanese government. Fujitsu’s revenue comes from sales of hardware such as computers, servers, and telecommunications gear, storage systems, software, and IT services.Read 3 remaining paragraphs | Comments
Categorías: Security Posts
Dell tells remote workers that they won’t be eligible for promotion
Enlarge (credit: Getty)
Starting in May, Dell employees who are fully remote will not be eligible for promotion, Business Insider (BI) reported Saturday. The upcoming policy update represents a dramatic reversal from Dell's prior stance on work from home (WFH), which included CEO Michael Dell saying: "If you are counting on forced hours spent in a traditional office to create collaboration and provide a feeling of belonging within your organization, you’re doing it wrong."
Dell employees will mostly all be considered "remote" or "hybrid" starting in May, BI reported. Hybrid workers have to come into the office at least 39 days per quarter, Dell confirmed to Ars Technica, which equates to approximately three times a week. Those who would prefer to never commute to an office will not "be considered for promotion, or be able to change roles," BI reported.
"For remote team members, it is important to understand the trade-offs: Career advancement, including applying to new roles in the company, will require a team member to reclassify as hybrid onsite," Dell's memo to workers said, per BI.Read 8 remaining paragraphs | Comments
Categorías: Security Posts
Elon Musk’s xAI releases Grok source and weights, taunting OpenAI
Enlarge / An AI-generated image released by xAI during the open-weights launch of Grok-1. (credit: xAI)
On Sunday, Elon Musk's AI firm xAI released the base model weights and network architecture of Grok-1, a large language model designed to compete with the models that power OpenAI's ChatGPT. The open-weights release through GitHub and BitTorrent comes as Musk continues to criticize (and sue) rival OpenAI for not releasing its AI models in an open way.
Announced in November, Grok is an AI assistant similar to ChatGPT that is available to X Premium+ subscribers who pay $16 a month to the social media platform formerly known as Twitter. At its heart is a mixture-of-experts LLM called "Grok-1," clocking in at 314 billion parameters. As a reference, GPT-3 included 175 billion parameters. Parameter count is a rough measure of an AI model's complexity, reflecting its potential for generating more useful responses.
xAI is releasing the base model of Grok-1, which is not fine-tuned for a specific task, so it is likely not the same model that X uses to power its Grok AI assistant. "This is the raw base model checkpoint from the Grok-1 pre-training phase, which concluded in October 2023," writes xAI on its release page. "This means that the model is not fine-tuned for any specific application, such as dialogue," meaning it's not necessarily shipping as a chatbot. But it will do next-token prediction, meaning it will complete a sentence (or other text prompt) with its estimation of the most relevant string of text.Read 9 remaining paragraphs | Comments
Categorías: Security Posts
Releasing the Attacknet: A new tool for finding bugs in blockchain nodes using chaos testing
By Benjamin Samuels (@thebensams)
Today, Trail of Bits is publishing Attacknet, a new tool that addresses the limitations of traditional runtime verification tools, built in collaboration with the Ethereum Foundation. Attacknet is intended to augment the EF’s current test methods by subjecting their execution and consensus clients to some of the most challenging network conditions imaginable.
Blockchain nodes must be held to the highest level of security assurance possible. Historically, the primary tools used to achieve this goal have been exhaustive specification, tests, client diversity, manual audits, and testnets. While these tools have traditionally done their job well, they collectively have serious limitations that can lead to critical bugs manifesting in a production environment, such as the May 2023 finality incident that occurred on Ethereum mainnet. Attacknet addresses these limitations by subjecting devnets to a much wider range of network conditions and misconfigurations than is possible on a conventional testnet.
How Attacknet works
Attacknet uses chaos engineering, a testing methodology that proactively injects faults into a production environment to verify that the system is tolerant to certain failures. These faults reproduce real-world problem scenarios and misconfigurations, and can be used to create exaggerated scenarios to test the boundary conditions of the blockchain.
Attacknet uses Chaos Mesh to inject faults into a devnet environment generated by Kurtosis. By building on top of Kurtosis and Chaos Mesh, Attacknet can create various network topologies with ensembles of different kinds of faults to push a blockchain network to its most extreme edge cases.
Some of the faults include:
- Clock skew, where a node’s clock is skewed forwards or backwards for a specific duration. Trail of Bits was able to reproduce the Ethereum finality incident using a clock skew fault, as detailed in our TrustX talk last year.
- Network latency, where a node’s connection to the network (or its corresponding EL/CL client) is delayed by a certain amount of time. This fault can help reproduce global latency conditions or help detect unintentional synchronicity assumptions in the blockchain’s consensus.
- Network partition, where the network is split into two or more halves that cannot communicate with each other. This fault can test the network’s fork choice rule, ability to re-org, and other edge cases.
- Network packet drop/corruption, where gossip packets are dropped or have their contents corrupted by a certain amount. This fault can test a node’s gossip validation and test the robustness of the network under hostile network conditions.
- Forced node crashes/offlining, where a certain client or type of client is ungracefully shut down. This fault can test the network’s resilience to validator inactivity, and test the ability of clients to re-sync to the network.
- I/O disk faults/latency, where a certain amount of latency or error rate is applied to all I/O operations a node makes. This fault can help profile nodes to understand their resource requirements, as I/O is often the largest limiting factor of node performance.
Categorías: Security Posts
Exploring the risks of eye-tracking technology in VR security
The content of this post is solely the responsibility of the author. AT&T does not adopt or endorse any of the views, positions, or information provided by the author in this article.
Virtual reality (VR) offers profound benefits across industries, particularly in education and training, thanks to its immersive nature. Through derivatives, such as 3D learning environments, VR enables learners to gain a deeper understanding of theoretical concepts more quickly and efficiently.
However, with the benefits come some dangers. One such risk is the integration of eye-tracking technology within virtual reality environments. While eye-tracking promises to make experiences better and improve security through biometric verification, it also raises privacy concerns.
This technology, though handy, could be exploited by cybercriminals. For instance, a recent paper by Rutgers University shows that hackers could use common virtual reality (AR/VR) headsets with motion sensors to capture facial movements linked to speech. This could lead to the theft of sensitive data communicated through voice commands, like credit card numbers and passwords.
This article explores the risks of this new technology, looking into how the information collected from our eyes could be misused and what it means for our security in virtual worlds.
How does VR eye-tracking work?
Eye-tracking technology in virtual reality (VR) is a sophisticated system designed to monitor and analyze where and how a user's gaze moves when they are immersed in a VR environment.
It achieves this through the use of infrared sensors and cameras embedded in the VR headset. These sensors aim infrared light toward the eyes, and the cameras capture the reflection of this light off the cornea and the position of the pupil. It then analyzes these reflections and positions to accurately determine the direction in which the user is looking.
Once the eye-tracking system gathers this data, it processes the information in real time, using sophisticated algorithms to interpret the user's gaze direction, eye movements, and other metrics such as pupil dilation and blink rate.
This comprehensive data allows the VR system to understand precisely where the user is focusing their attention within the virtual environment.
At the rate at which VR technology is growing, most people instantly think of monitoring and data selling, but also, at the same time, it’s not all doom and gloom. We might be moving towards a futuristic workplace, where we can focus on creative aspects of our job. Imagine a developer being able to receive suggestions about cloud cost optimization or writing cleaner, more readable code. Still, the concerns are yet to be addressed.
Privacy concerns with eye-tracking technology
Don’t get us wrong—eye-tracking technology can have many benefits. For instance, it has been used to identify cognitive disorders such as autism and attention deficit disorder, as well as mental and psychological illnesses like schizophrenia and Alzheimer's. It can also provide insights into a person's behavior, including potential indicators of drug and alcohol use.
The data that it collects sometimes can also go beyond just where an individual is looking, and it’s been one of the main issues surrounding VR games. While the notion of monetizing eye-tracking data is still a theoretical one, there’s a lot that companies can infer from it.
This capability extends to understanding which advertisements catch our attention, how we process information on a webpage, and our reactions to various stimuli. While it may seem great to have your VR headset track your activity in the game and serve you the best suggestions to buy a WordPress plugin, provide you with ideas for your domain name, or use AI to generate helpful answers, the true possibilities are much more sinister.
Thus, safeguarding this data through robust privacy policies and data-centric security practices is essential to mitigate the risks associated with its misuse. As eye-tracking devices are starting to parallel the ubiquity of webcams, regulators must stay ahead of data-hungry corporations.
Potential for misuse of eye-tracking data
Eye-tracking technology, while innovative and rich in potential for enhancing user experiences in various fields, including VR, also harbors significant risks regarding data privacy and security.
The detailed data captured by eye-tracking — ranging from where individuals look, and how long they gaze at specific points to more subtle metrics like pupil dilation — can reveal an enormous amount about a person's preferences, interests, and even their emotional or psychological state.
This raises a significant ethical dilemma: What if companies like Google suddenly begin collecting and storing data on users' eye movements? This could pose a problem for organizations planning to adopt VR technology in the future, especially those handling sensitive data.
With an ever-more privacy-aware consumer base, they might even be compelled to look for a GCP alternative, different email hosting providers, and a host of other solutions to protect their users' privacy and adapt to their preferences.
The potential risks of eye-tracking data misuse are vast and varied — here is a concise overview of some of the more pressing issues:
- Personal profiling. Eye-tracking data can be used to construct detailed profiles of users, including their interests, habits, and behaviors. This information could potentially be exploited for targeted advertising in a way that infringes on personal privacy.
- Surveillance. In the wrong hands, eye-tracking data could serve as a tool for surveillance, allowing unauthorized tracking of an individual's focus and attention in both digital and physical spaces.
- Manipulation and influence. Figuring out what captures a person's attention or triggers emotional responses could give other people or organizations the power to manipulate decisions. Imagine WordPress taping into its database of 455 million websites and using eye-tracking data to suggest plugins and other products to those they think will be more likely to purchase them.
- Security breaches. Like any digital data, eye-tracking information is susceptible to hacking and unauthorized access. If such data were compromised, it could lead to identity theft, blackmail, or other forms of cybercrime, particularly if combined with other personal data.
- Unintended inferences. Eye-tracking could inadvertently expose sensitive information about a person's health (e.g., detecting conditions like Parkinson's or Alzheimer's disease based on eye movement patterns) or other personal attributes without their consent.
Categorías: Security Posts
Cybersecurity Concerns for Ancillary Strength Control Subsystems
Additive manufacturing (AM) engineers have been incredibly creative in developing ancillary systems that modify a printed parts mechanical properties. These systems mostly focus on the issue of anisotropic properties of additively built components. This blog post is a good reference if you are unfamiliar with isotropic vs anisotropic properties and how they impact 3d printing. […]
The post Cybersecurity Concerns for Ancillary Strength Control Subsystems appeared first on BreakPoint Labs - Blog.
Categorías: Security Posts
Update on Naked Security
To consolidate all of our security intelligence and news in one location, we have migrated Naked Security to the Sophos News platform.
Categorías: Security Posts