1
0
mirror of synced 2024-11-24 11:43:12 +01:00

Added support for the Obsolete attribute to remove bindings and queues with backwards compatibility

Updated license in nuspec to comply with the new specifications
This commit is contained in:
Mark van Renswoude 2019-08-20 11:47:53 +02:00
parent 295b584969
commit bef3961f7f
25 changed files with 336 additions and 88 deletions

View File

@ -61,6 +61,11 @@ namespace ExampleLib
public async Task WaitAsync() public async Task WaitAsync()
{ {
await doneSignal.Task; await doneSignal.Task;
// This is a hack, because the signal is often given in a message handler before the message can be
// acknowledged, causing it to be put back on the queue because the connection is closed.
// This short delay allows consumers to finish. This is not an issue in a proper service application.
await Task.Delay(500);
} }

View File

@ -6,7 +6,7 @@
<title>Tapeti Annotations</title> <title>Tapeti Annotations</title>
<authors>Mark van Renswoude</authors> <authors>Mark van Renswoude</authors>
<owners>Mark van Renswoude</owners> <owners>Mark van Renswoude</owners>
<licenseUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/UNLICENSE</licenseUrl> <license type="expression">Unlicense</license>
<projectUrl>https://github.com/MvRens/Tapeti</projectUrl> <projectUrl>https://github.com/MvRens/Tapeti</projectUrl>
<iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.Annotations.png</iconUrl> <iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.Annotations.png</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance> <requireLicenseAcceptance>false</requireLicenseAcceptance>

View File

@ -6,7 +6,7 @@
<title>Tapeti Autofac</title> <title>Tapeti Autofac</title>
<authors>Mark van Renswoude</authors> <authors>Mark van Renswoude</authors>
<owners>Mark van Renswoude</owners> <owners>Mark van Renswoude</owners>
<licenseUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/UNLICENSE</licenseUrl> <license type="expression">Unlicense</license>
<projectUrl>https://github.com/MvRens/Tapeti</projectUrl> <projectUrl>https://github.com/MvRens/Tapeti</projectUrl>
<iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.SimpleInjector.png</iconUrl> <iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.SimpleInjector.png</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance> <requireLicenseAcceptance>false</requireLicenseAcceptance>

View File

@ -6,7 +6,7 @@
<title>Tapeti Castle Windsor</title> <title>Tapeti Castle Windsor</title>
<authors>Mark van Renswoude</authors> <authors>Mark van Renswoude</authors>
<owners>Mark van Renswoude</owners> <owners>Mark van Renswoude</owners>
<licenseUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/UNLICENSE</licenseUrl> <license type="expression">Unlicense</license>
<projectUrl>https://github.com/MvRens/Tapeti</projectUrl> <projectUrl>https://github.com/MvRens/Tapeti</projectUrl>
<iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.SimpleInjector.png</iconUrl> <iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.SimpleInjector.png</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance> <requireLicenseAcceptance>false</requireLicenseAcceptance>

View File

@ -6,7 +6,7 @@
<title>Tapeti DataAnnotations Extensions</title> <title>Tapeti DataAnnotations Extensions</title>
<authors>Mark van Renswoude</authors> <authors>Mark van Renswoude</authors>
<owners>Mark van Renswoude</owners> <owners>Mark van Renswoude</owners>
<licenseUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/UNLICENSE</licenseUrl> <license type="expression">Unlicense</license>
<projectUrl>https://github.com/MvRens/Tapeti</projectUrl> <projectUrl>https://github.com/MvRens/Tapeti</projectUrl>
<iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.DataAnnotations.png</iconUrl> <iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.DataAnnotations.png</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance> <requireLicenseAcceptance>false</requireLicenseAcceptance>

View File

@ -6,7 +6,7 @@
<title>Tapeti DataAnnotations</title> <title>Tapeti DataAnnotations</title>
<authors>Mark van Renswoude</authors> <authors>Mark van Renswoude</authors>
<owners>Mark van Renswoude</owners> <owners>Mark van Renswoude</owners>
<licenseUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/UNLICENSE</licenseUrl> <license type="expression">Unlicense</license>
<projectUrl>https://github.com/MvRens/Tapeti</projectUrl> <projectUrl>https://github.com/MvRens/Tapeti</projectUrl>
<iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.DataAnnotations.png</iconUrl> <iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.DataAnnotations.png</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance> <requireLicenseAcceptance>false</requireLicenseAcceptance>

View File

@ -6,7 +6,7 @@
<title>Tapeti Flow SQL</title> <title>Tapeti Flow SQL</title>
<authors>Mark van Renswoude</authors> <authors>Mark van Renswoude</authors>
<owners>Mark van Renswoude</owners> <owners>Mark van Renswoude</owners>
<licenseUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/UNLICENSE</licenseUrl> <license type="expression">Unlicense</license>
<projectUrl>https://github.com/MvRens/Tapeti</projectUrl> <projectUrl>https://github.com/MvRens/Tapeti</projectUrl>
<iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.Flow.SQL.png</iconUrl> <iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.Flow.SQL.png</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance> <requireLicenseAcceptance>false</requireLicenseAcceptance>

View File

@ -6,7 +6,7 @@
<title>Tapeti Flow</title> <title>Tapeti Flow</title>
<authors>Menno van Lavieren, Mark van Renswoude</authors> <authors>Menno van Lavieren, Mark van Renswoude</authors>
<owners>Mark van Renswoude</owners> <owners>Mark van Renswoude</owners>
<licenseUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/UNLICENSE</licenseUrl> <license type="expression">Unlicense</license>
<projectUrl>https://github.com/MvRens/Tapeti</projectUrl> <projectUrl>https://github.com/MvRens/Tapeti</projectUrl>
<iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.Flow.png</iconUrl> <iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.Flow.png</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance> <requireLicenseAcceptance>false</requireLicenseAcceptance>

View File

@ -6,7 +6,7 @@
<title>Tapeti Ninject</title> <title>Tapeti Ninject</title>
<authors>Mark van Renswoude</authors> <authors>Mark van Renswoude</authors>
<owners>Mark van Renswoude</owners> <owners>Mark van Renswoude</owners>
<licenseUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/UNLICENSE</licenseUrl> <license type="expression">Unlicense</license>
<projectUrl>https://github.com/MvRens/Tapeti</projectUrl> <projectUrl>https://github.com/MvRens/Tapeti</projectUrl>
<iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.SimpleInjector.png</iconUrl> <iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.SimpleInjector.png</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance> <requireLicenseAcceptance>false</requireLicenseAcceptance>

View File

@ -6,7 +6,7 @@
<title>Tapeti Serilog</title> <title>Tapeti Serilog</title>
<authors>Hans Mulder</authors> <authors>Hans Mulder</authors>
<owners>Hans Mulder</owners> <owners>Hans Mulder</owners>
<licenseUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/UNLICENSE</licenseUrl> <license type="expression">Unlicense</license>
<projectUrl>https://github.com/MvRens/Tapeti</projectUrl> <projectUrl>https://github.com/MvRens/Tapeti</projectUrl>
<iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.Serilog.png</iconUrl> <iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.Serilog.png</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance> <requireLicenseAcceptance>false</requireLicenseAcceptance>

View File

@ -71,5 +71,14 @@ namespace Tapeti.Serilog
contextLogger.Error(exception, "Tapeti: exception in message handler"); contextLogger.Error(exception, "Tapeti: exception in message handler");
} }
/// <inheritdoc />
public void QueueObsolete(string queueName, bool deleted, uint messageCount)
{
if (deleted)
seriLogger.Information("Tapeti: obsolete queue {queue} has been deleted", queueName);
else
seriLogger.Information("Tapeti: obsolete queue {queue} has been unbound but not yet deleted, {messageCount} messages remaining", queueName, messageCount);
}
} }
} }

View File

@ -6,7 +6,7 @@
<title>Tapeti SimpleInjector</title> <title>Tapeti SimpleInjector</title>
<authors>Mark van Renswoude</authors> <authors>Mark van Renswoude</authors>
<owners>Mark van Renswoude</owners> <owners>Mark van Renswoude</owners>
<licenseUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/UNLICENSE</licenseUrl> <license type="expression">Unlicense</license>
<projectUrl>https://github.com/MvRens/Tapeti</projectUrl> <projectUrl>https://github.com/MvRens/Tapeti</projectUrl>
<iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.SimpleInjector.png</iconUrl> <iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.SimpleInjector.png</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance> <requireLicenseAcceptance>false</requireLicenseAcceptance>

View File

@ -6,7 +6,7 @@
<title>Tapeti Transient</title> <title>Tapeti Transient</title>
<authors>Menno van Lavieren, Mark van Renswoude</authors> <authors>Menno van Lavieren, Mark van Renswoude</authors>
<owners>Mark van Renswoude</owners> <owners>Mark van Renswoude</owners>
<licenseUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/UNLICENSE</licenseUrl> <license type="expression">Unlicense</license>
<projectUrl>https://github.com/MvRens/Tapeti</projectUrl> <projectUrl>https://github.com/MvRens/Tapeti</projectUrl>
<iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.Flow.png</iconUrl> <iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.Flow.png</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance> <requireLicenseAcceptance>false</requireLicenseAcceptance>

View File

@ -6,7 +6,7 @@
<title>Tapeti UnityContainer</title> <title>Tapeti UnityContainer</title>
<authors>Mark van Renswoude</authors> <authors>Mark van Renswoude</authors>
<owners>Mark van Renswoude</owners> <owners>Mark van Renswoude</owners>
<licenseUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/UNLICENSE</licenseUrl> <license type="expression">Unlicense</license>
<projectUrl>https://github.com/MvRens/Tapeti</projectUrl> <projectUrl>https://github.com/MvRens/Tapeti</projectUrl>
<iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.SimpleInjector.png</iconUrl> <iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.SimpleInjector.png</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance> <requireLicenseAcceptance>false</requireLicenseAcceptance>

View File

@ -117,5 +117,12 @@ namespace Tapeti.Config
/// <param name="queuePrefix">An optional prefix for the dynamic queue's name. If not provided, RabbitMQ's default logic will be used to create an amq.gen queue.</param> /// <param name="queuePrefix">An optional prefix for the dynamic queue's name. If not provided, RabbitMQ's default logic will be used to create an amq.gen queue.</param>
/// <returns>The generated name of the dynamic queue</returns> /// <returns>The generated name of the dynamic queue</returns>
Task<string> BindDynamicDirect(string queuePrefix = null); Task<string> BindDynamicDirect(string queuePrefix = null);
/// <summary>
/// Marks the specified durable queue as having an obsolete binding. If after all bindings have subscribed, the queue only contains obsolete
/// bindings and is empty, it will be removed.
/// </summary>
/// <param name="queueName">The name of the durable queue</param>
Task BindDurableObsolete(string queueName);
} }
} }

View File

@ -91,6 +91,13 @@ namespace Tapeti.Connection
/// <param name="queueName">The name of the queue to verify</param> /// <param name="queueName">The name of the queue to verify</param>
Task DurableQueueVerify(string queueName); Task DurableQueueVerify(string queueName);
/// <summary>
/// Deletes a durable queue.
/// </summary>
/// <param name="queueName">The name of the queue to delete</param>
/// <param name="onlyIfEmpty">If true, the queue will only be deleted if it is empty otherwise all bindings will be removed. If false, the queue is deleted even if there are queued messages.</param>
Task DurableQueueDelete(string queueName, bool onlyIfEmpty = true);
/// <summary> /// <summary>
/// Creates a dynamic queue. /// Creates a dynamic queue.
/// </summary> /// </summary>

View File

@ -50,6 +50,7 @@ namespace Tapeti.Connection
private ulong lastDeliveryTag; private ulong lastDeliveryTag;
private DateTime connectedDateTime; private DateTime connectedDateTime;
private readonly HttpClient managementClient; private readonly HttpClient managementClient;
private readonly HashSet<string> deletedQueues = new HashSet<string>();
// These fields must be locked, since the callbacks for BasicAck/BasicReturn can run in a different thread // These fields must be locked, since the callbacks for BasicAck/BasicReturn can run in a different thread
private readonly object confirmLock = new object(); private readonly object confirmLock = new object();
@ -185,16 +186,16 @@ namespace Tapeti.Connection
/// <inheritdoc /> /// <inheritdoc />
public async Task Consume(string queueName, IConsumer consumer) public async Task Consume(string queueName, IConsumer consumer)
{ {
if (deletedQueues.Contains(queueName))
return;
if (string.IsNullOrEmpty(queueName)) if (string.IsNullOrEmpty(queueName))
throw new ArgumentNullException(nameof(queueName)); throw new ArgumentNullException(nameof(queueName));
await taskQueue.Value.Add(() => await QueueWithRetryableChannel(channel =>
{ {
WithRetryableChannel(channel => var basicConsumer = new TapetiBasicConsumer(consumer, Respond);
{ channel.BasicConsume(queueName, false, basicConsumer);
var basicConsumer = new TapetiBasicConsumer(consumer, Respond);
channel.BasicConsume(queueName, false, basicConsumer);
});
}); });
} }
@ -223,7 +224,6 @@ namespace Tapeti.Connection
default: default:
throw new ArgumentOutOfRangeException(nameof(result), result, null); throw new ArgumentOutOfRangeException(nameof(result), result, null);
} }
}); });
} }
@ -255,32 +255,106 @@ namespace Tapeti.Connection
/// <inheritdoc /> /// <inheritdoc />
public async Task DurableQueueVerify(string queueName) public async Task DurableQueueVerify(string queueName)
{ {
await taskQueue.Value.Add(() => await QueueWithRetryableChannel(channel =>
{ {
WithRetryableChannel(channel => channel.QueueDeclarePassive(queueName);
{
channel.QueueDeclarePassive(queueName);
});
}); });
} }
/// <inheritdoc />
public async Task DurableQueueDelete(string queueName, bool onlyIfEmpty = true)
{
if (!onlyIfEmpty)
{
uint deletedMessages = 0;
await QueueWithRetryableChannel(channel =>
{
deletedMessages = channel.QueueDelete(queueName);
});
deletedQueues.Add(queueName);
logger.QueueObsolete(queueName, true, deletedMessages);
return;
}
await taskQueue.Value.Add(async () =>
{
bool retry;
do
{
retry = false;
// Get queue information from the Management API, since the AMQP operations will
// throw an error if the queue does not exist or still contains messages and resets
// the connection. The resulting reconnect will cause subscribers to reset.
var queueInfo = await GetQueueInfo(queueName);
if (queueInfo == null)
{
deletedQueues.Add(queueName);
return;
}
if (queueInfo.Messages == 0)
{
// Still pass onlyIfEmpty to prevent concurrency issues if a message arrived between
// the call to the Management API and deleting the queue. Because the QueueWithRetryableChannel
// includes the GetQueueInfo, the next time around it should have Messages > 0
try
{
WithRetryableChannel(channel =>
{
channel.QueueDelete(queueName, false, true);
});
deletedQueues.Add(queueName);
logger.QueueObsolete(queueName, true, 0);
}
catch (OperationInterruptedException e)
{
if (e.ShutdownReason.ReplyCode == RabbitMQ.Client.Framing.Constants.PreconditionFailed)
retry = true;
else
throw;
}
}
else
{
// Remove all bindings instead
var existingBindings = (await GetQueueBindings(queueName)).ToList();
if (existingBindings.Count > 0)
{
WithRetryableChannel(channel =>
{
foreach (var binding in existingBindings)
channel.QueueUnbind(queueName, binding.Exchange, binding.RoutingKey);
});
}
logger.QueueObsolete(queueName, false, queueInfo.Messages);
}
} while (retry);
});
}
/// <inheritdoc /> /// <inheritdoc />
public async Task<string> DynamicQueueDeclare(string queuePrefix = null) public async Task<string> DynamicQueueDeclare(string queuePrefix = null)
{ {
string queueName = null; string queueName = null;
await taskQueue.Value.Add(() => await QueueWithRetryableChannel(channel =>
{ {
WithRetryableChannel(channel => if (!string.IsNullOrEmpty(queuePrefix))
{ {
if (!string.IsNullOrEmpty(queuePrefix)) queueName = queuePrefix + "." + Guid.NewGuid().ToString("N");
{ channel.QueueDeclare(queueName);
queueName = queuePrefix + "." + Guid.NewGuid().ToString("N"); }
channel.QueueDeclare(queueName); else
} queueName = channel.QueueDeclare().QueueName;
else
queueName = channel.QueueDeclare().QueueName;
});
}); });
return queueName; return queueName;
@ -289,13 +363,10 @@ namespace Tapeti.Connection
/// <inheritdoc /> /// <inheritdoc />
public async Task DynamicQueueBind(string queueName, QueueBinding binding) public async Task DynamicQueueBind(string queueName, QueueBinding binding)
{ {
await taskQueue.Value.Add(() => await QueueWithRetryableChannel(channel =>
{ {
WithRetryableChannel(channel => DeclareExchange(channel, binding.Exchange);
{ channel.QueueBind(queueName, binding.Exchange, binding.RoutingKey);
DeclareExchange(channel, binding.Exchange);
channel.QueueBind(queueName, binding.Exchange, binding.RoutingKey);
});
}); });
} }
@ -335,18 +406,31 @@ namespace Tapeti.Connection
HttpStatusCode.ServiceUnavailable HttpStatusCode.ServiceUnavailable
}; };
private static readonly TimeSpan[] ExponentialBackoff =
private class ManagementQueueInfo
{ {
TimeSpan.FromSeconds(1), [JsonProperty("messages")]
TimeSpan.FromSeconds(2), public uint Messages { get; set; }
TimeSpan.FromSeconds(3), }
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(8),
TimeSpan.FromSeconds(13),
TimeSpan.FromSeconds(21), private async Task<ManagementQueueInfo> GetQueueInfo(string queueName)
TimeSpan.FromSeconds(34), {
TimeSpan.FromSeconds(55) var virtualHostPath = Uri.EscapeDataString(connectionParams.VirtualHost);
}; var queuePath = Uri.EscapeDataString(queueName);
return await WithRetryableManagementAPI($"queues/{virtualHostPath}/{queuePath}", async response =>
{
if (response.StatusCode == HttpStatusCode.NotFound)
return null;
response.EnsureSuccessStatusCode();
var content = await response.Content.ReadAsStringAsync();
return JsonConvert.DeserializeObject<ManagementQueueInfo>(content);
});
}
private class ManagementBinding private class ManagementBinding
@ -378,10 +462,42 @@ namespace Tapeti.Connection
{ {
var virtualHostPath = Uri.EscapeDataString(connectionParams.VirtualHost); var virtualHostPath = Uri.EscapeDataString(connectionParams.VirtualHost);
var queuePath = Uri.EscapeDataString(queueName); var queuePath = Uri.EscapeDataString(queueName);
var requestUri = new Uri($"http://{connectionParams.HostName}:{connectionParams.ManagementPort}/api/queues/{virtualHostPath}/{queuePath}/bindings");
return await WithRetryableManagementAPI($"queues/{virtualHostPath}/{queuePath}/bindings", async response =>
{
response.EnsureSuccessStatusCode();
var content = await response.Content.ReadAsStringAsync();
var bindings = JsonConvert.DeserializeObject<IEnumerable<ManagementBinding>>(content);
// Filter out the binding to an empty source, which is always present for direct-to-queue routing
return bindings
.Where(binding => !string.IsNullOrEmpty(binding.Source))
.Select(binding => new QueueBinding(binding.Source, binding.RoutingKey));
});
}
private static readonly TimeSpan[] ExponentialBackoff =
{
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(3),
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(8),
TimeSpan.FromSeconds(13),
TimeSpan.FromSeconds(21),
TimeSpan.FromSeconds(34),
TimeSpan.FromSeconds(55)
};
private async Task<T> WithRetryableManagementAPI<T>(string path, Func<HttpResponseMessage, Task<T>> handleResponse)
{
var requestUri = new Uri($"http://{connectionParams.HostName}:{connectionParams.ManagementPort}/api/{path}");
using (var request = new HttpRequestMessage(HttpMethod.Get, requestUri)) using (var request = new HttpRequestMessage(HttpMethod.Get, requestUri))
{ {
var retryDelayIndex = 0; var retryDelayIndex = 0;
while (true) while (true)
@ -389,15 +505,7 @@ namespace Tapeti.Connection
try try
{ {
var response = await managementClient.SendAsync(request); var response = await managementClient.SendAsync(request);
response.EnsureSuccessStatusCode(); return await handleResponse(response);
var content = await response.Content.ReadAsStringAsync();
var bindings = JsonConvert.DeserializeObject<IEnumerable<ManagementBinding>>(content);
// Filter out the binding to an empty source, which is always present for direct-to-queue routing
return bindings
.Where(binding => !string.IsNullOrEmpty(binding.Source))
.Select(binding => new QueueBinding(binding.Source, binding.RoutingKey));
} }
catch (TimeoutException) catch (TimeoutException)
{ {
@ -435,6 +543,15 @@ namespace Tapeti.Connection
} }
private async Task QueueWithRetryableChannel(Action<IModel> operation)
{
await taskQueue.Value.Add(() =>
{
WithRetryableChannel(operation);
});
}
/// <remarks> /// <remarks>
/// Only call this from a task in the taskQueue to ensure IModel is only used /// Only call this from a task in the taskQueue to ensure IModel is only used
/// by a single thread, as is recommended in the RabbitMQ .NET Client documentation. /// by a single thread, as is recommended in the RabbitMQ .NET Client documentation.

View File

@ -91,6 +91,7 @@ namespace Tapeti.Connection
public abstract Task BindDurable(Type messageClass, string queueName); public abstract Task BindDurable(Type messageClass, string queueName);
public abstract Task BindDurableDirect(string queueName); public abstract Task BindDurableDirect(string queueName);
public abstract Task BindDurableObsolete(string queueName);
public async Task<string> BindDynamic(Type messageClass, string queuePrefix = null) public async Task<string> BindDynamic(Type messageClass, string queuePrefix = null)
@ -182,6 +183,7 @@ namespace Tapeti.Connection
private class DeclareDurableQueuesBindingTarget : CustomBindingTarget private class DeclareDurableQueuesBindingTarget : CustomBindingTarget
{ {
private readonly Dictionary<string, List<Type>> durableQueues = new Dictionary<string, List<Type>>(); private readonly Dictionary<string, List<Type>> durableQueues = new Dictionary<string, List<Type>>();
private readonly HashSet<string> obsoleteDurableQueues = new HashSet<string>();
public DeclareDurableQueuesBindingTarget(Func<ITapetiClient> clientFactory, IRoutingKeyStrategy routingKeyStrategy, IExchangeStrategy exchangeStrategy) : base(clientFactory, routingKeyStrategy, exchangeStrategy) public DeclareDurableQueuesBindingTarget(Func<ITapetiClient> clientFactory, IRoutingKeyStrategy routingKeyStrategy, IExchangeStrategy exchangeStrategy) : base(clientFactory, routingKeyStrategy, exchangeStrategy)
@ -217,10 +219,23 @@ namespace Tapeti.Connection
} }
public override Task BindDurableObsolete(string queueName)
{
obsoleteDurableQueues.Add(queueName);
return Task.CompletedTask;
}
public override async Task Apply() public override async Task Apply()
{ {
var worker = ClientFactory(); var worker = ClientFactory();
await DeclareQueues(worker);
await DeleteObsoleteQueues(worker);
}
private async Task DeclareQueues(ITapetiClient worker)
{
await Task.WhenAll(durableQueues.Select(async queue => await Task.WhenAll(durableQueues.Select(async queue =>
{ {
var bindings = queue.Value.Select(messageClass => var bindings = queue.Value.Select(messageClass =>
@ -234,6 +249,15 @@ namespace Tapeti.Connection
await worker.DurableQueueDeclare(queue.Key, bindings); await worker.DurableQueueDeclare(queue.Key, bindings);
})); }));
} }
private async Task DeleteObsoleteQueues(ITapetiClient worker)
{
await Task.WhenAll(obsoleteDurableQueues.Except(durableQueues.Keys).Select(async queue =>
{
await worker.DurableQueueDelete(queue);
}));
}
} }
@ -257,6 +281,11 @@ namespace Tapeti.Connection
await VerifyDurableQueue(queueName); await VerifyDurableQueue(queueName);
} }
public override Task BindDurableObsolete(string queueName)
{
return Task.CompletedTask;
}
private async Task VerifyDurableQueue(string queueName) private async Task VerifyDurableQueue(string queueName)
{ {

View File

@ -45,5 +45,13 @@ namespace Tapeti.Default
Console.WriteLine(); Console.WriteLine();
Console.WriteLine(exception); Console.WriteLine(exception);
} }
/// <inheritdoc />
public void QueueObsolete(string queueName, bool deleted, uint messageCount)
{
Console.WriteLine(deleted
? $"[Tapeti] Obsolete queue was deleted: {queueName}"
: $"[Tapeti] Obsolete queue bindings removed: {queueName}, {messageCount} messages remaining");
}
} }
} }

View File

@ -46,6 +46,12 @@ namespace Tapeti.Default
/// </summary> /// </summary>
public BindingTargetMode BindingTargetMode; public BindingTargetMode BindingTargetMode;
/// <summary>
/// Indicates if the method or controller is marked with the Obsolete attribute, indicating it should
/// only handle messages already in the queue and not bind to the routing key for new messages.
/// </summary>
public bool IsObsolete;
/// <summary> /// <summary>
/// Value factories for the method parameters. /// Value factories for the method parameters.
/// </summary> /// </summary>
@ -106,32 +112,40 @@ namespace Tapeti.Default
/// <inheritdoc /> /// <inheritdoc />
public async Task Apply(IBindingTarget target) public async Task Apply(IBindingTarget target)
{ {
switch (bindingInfo.BindingTargetMode) if (!bindingInfo.IsObsolete)
{ {
case BindingTargetMode.Default: switch (bindingInfo.BindingTargetMode)
if (bindingInfo.QueueInfo.QueueType == QueueType.Dynamic) {
QueueName = await target.BindDynamic(bindingInfo.MessageClass, bindingInfo.QueueInfo.Name); case BindingTargetMode.Default:
else if (bindingInfo.QueueInfo.QueueType == QueueType.Dynamic)
{ QueueName = await target.BindDynamic(bindingInfo.MessageClass, bindingInfo.QueueInfo.Name);
await target.BindDurable(bindingInfo.MessageClass, bindingInfo.QueueInfo.Name); else
QueueName = bindingInfo.QueueInfo.Name; {
} await target.BindDurable(bindingInfo.MessageClass, bindingInfo.QueueInfo.Name);
QueueName = bindingInfo.QueueInfo.Name;
}
break; break;
case BindingTargetMode.Direct: case BindingTargetMode.Direct:
if (bindingInfo.QueueInfo.QueueType == QueueType.Dynamic) if (bindingInfo.QueueInfo.QueueType == QueueType.Dynamic)
QueueName = await target.BindDynamicDirect(bindingInfo.MessageClass, bindingInfo.QueueInfo.Name); QueueName = await target.BindDynamicDirect(bindingInfo.MessageClass, bindingInfo.QueueInfo.Name);
else else
{ {
await target.BindDurableDirect(bindingInfo.QueueInfo.Name); await target.BindDurableDirect(bindingInfo.QueueInfo.Name);
QueueName = bindingInfo.QueueInfo.Name; QueueName = bindingInfo.QueueInfo.Name;
} }
break; break;
default: default:
throw new ArgumentOutOfRangeException(nameof(bindingInfo.BindingTargetMode), bindingInfo.BindingTargetMode, "Invalid BindingTargetMode"); throw new ArgumentOutOfRangeException(nameof(bindingInfo.BindingTargetMode), bindingInfo.BindingTargetMode, "Invalid BindingTargetMode");
}
}
else if (bindingInfo.QueueInfo.QueueType == QueueType.Durable)
{
await target.BindDurableObsolete(bindingInfo.QueueInfo.Name);
QueueName = bindingInfo.QueueInfo.Name;
} }
} }

View File

@ -28,5 +28,10 @@ namespace Tapeti.Default
public void ConsumeException(Exception exception, IMessageContext messageContext, ConsumeResult consumeResult) public void ConsumeException(Exception exception, IMessageContext messageContext, ConsumeResult consumeResult)
{ {
} }
/// <inheritdoc />
public void QueueObsolete(string queueName, bool deleted, uint messageCount)
{
}
} }
} }

View File

@ -42,5 +42,13 @@ namespace Tapeti
/// <param name="messageContext"></param> /// <param name="messageContext"></param>
/// <param name="consumeResult">Indicates the action taken by the exception handler</param> /// <param name="consumeResult">Indicates the action taken by the exception handler</param>
void ConsumeException(Exception exception, IMessageContext messageContext, ConsumeResult consumeResult); void ConsumeException(Exception exception, IMessageContext messageContext, ConsumeResult consumeResult);
/// <summary>
/// Called when a queue is determined to be obsolete.
/// </summary>
/// <param name="queueName"></param>
/// <param name="deleted">True if the queue was empty and has been deleted, false if there are still messages to process</param>
/// <param name="messageCount">If deleted, the number of messages purged, otherwise the number of messages still in the queue</param>
void QueueObsolete(string queueName, bool deleted, uint messageCount);
} }
} }

View File

@ -6,7 +6,7 @@
<title>Tapeti</title> <title>Tapeti</title>
<authors>Mark van Renswoude</authors> <authors>Mark van Renswoude</authors>
<owners>Mark van Renswoude</owners> <owners>Mark van Renswoude</owners>
<licenseUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/UNLICENSE</licenseUrl> <license type="expression">Unlicense</license>
<projectUrl>https://github.com/MvRens/Tapeti</projectUrl> <projectUrl>https://github.com/MvRens/Tapeti</projectUrl>
<iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.png</iconUrl> <iconUrl>https://raw.githubusercontent.com/MvRens/Tapeti/master/resources/icons/Tapeti.png</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance> <requireLicenseAcceptance>false</requireLicenseAcceptance>

View File

@ -40,6 +40,9 @@ namespace Tapeti
var controllerQueueInfo = GetQueueInfo(controller); var controllerQueueInfo = GetQueueInfo(controller);
(builderAccess.DependencyResolver as IDependencyContainer)?.RegisterController(controller); (builderAccess.DependencyResolver as IDependencyContainer)?.RegisterController(controller);
var controllerIsObsolete = controller.GetCustomAttribute<ObsoleteAttribute>() != null;
foreach (var method in controller.GetMembers(BindingFlags.Public | BindingFlags.Instance) foreach (var method in controller.GetMembers(BindingFlags.Public | BindingFlags.Instance)
.Where(m => m.MemberType == MemberTypes.Method && m.DeclaringType != typeof(object) && (m as MethodInfo)?.IsSpecialName == false) .Where(m => m.MemberType == MemberTypes.Method && m.DeclaringType != typeof(object) && (m as MethodInfo)?.IsSpecialName == false)
.Select(m => (MethodInfo)m)) .Select(m => (MethodInfo)m))
@ -50,6 +53,9 @@ namespace Tapeti
$"Method {method.Name} or controller {controller.Name} requires a queue attribute"); $"Method {method.Name} or controller {controller.Name} requires a queue attribute");
var methodIsObsolete = controllerIsObsolete || method.GetCustomAttribute<ObsoleteAttribute>() != null;
var context = new ControllerBindingContext(method.GetParameters(), method.ReturnParameter) var context = new ControllerBindingContext(method.GetParameters(), method.ReturnParameter)
{ {
Controller = controller, Controller = controller,
@ -83,6 +89,7 @@ namespace Tapeti
QueueInfo = methodQueueInfo, QueueInfo = methodQueueInfo,
MessageClass = context.MessageClass, MessageClass = context.MessageClass,
BindingTargetMode = context.BindingTargetMode, BindingTargetMode = context.BindingTargetMode,
IsObsolete = methodIsObsolete,
ParameterFactories = context.GetParameterHandlers(), ParameterFactories = context.GetParameterHandlers(),
ResultHandler = context.GetResultHandler(), ResultHandler = context.GetResultHandler(),

View File

@ -45,9 +45,41 @@ To enable the automatic creation of durable queues, call EnableDeclareDurableQue
.Build(); .Build();
The queue will be bound to all message classes for which you have defined a message handler. If the queue already existed and contains bindings which are no longer valid, those bindings will be removed. Note however that if there are still messages of that type in the queue they will be consumed and cause an exception. The queue will be bound to all message classes for which you have defined a message handler. If the queue already existed and contains bindings which are no longer valid, those bindings will be removed. Note however that if there are still messages of that type in the queue they will be consumed and cause an exception. To keep the queue backwards compatible, see the next section on migrating durable queues.
At the time of writing there is no special support for obsolete queues. Once a durable queue is no longer referenced in the service it will remain in RabbitMQ along with any messages in it, without a consumer. This allows you to inspect the contents, perform any migrating steps necessary and delete the queue manually.
Migrating durable queues
------------------------
.. note:: This section assumes you are using EnableDeclareDurableQueues.
As your service evolves so can your message handlers. Perhaps a message no longer needs to handled, or you want to split them into another queue.
If you remove a message handler the binding will also be removed from the queue, but there may still be messages of that type in the queue. Since these have nowhere to go, they will cause an error and be lost.
Instead of removing the message handler you can mark it with the standard .NET ``[Obsolete]`` attribute:
::
[MessageController]
[DurableQueue("monitoring")]
public class ObsoleteMonitoringController
{
[Obsolete]
public void HandleEscapeMessage(RabbitEscapedMessage message)
{
// Handle the message like before, perform the necessary migration,
// or simply ignore it if you no longer need it.
}
}
Messages will still be consumed from the queue as long as it exists, but the routing key binding will removed so no new messages of that type will be delivered.
The ``[Obsolete]`` attribute can also be applied to the entire controller to mark all message handlers it contains as obsolete.
If all message handlers bound to a durable queue are marked as obsolete, including other controllers bound to the same durable queue, the queue is a candidate for removal. During startup, if the queue is empty it will be deleted. This action is logged to the registered ILogger.
If there are still messages in the queue it's pending removal will be logged but the consumers will run as normal to empty the queue. The queue will then remain until it is checked again when the application is restarted.
Request - response Request - response