TechOps Toolkit

MX Support 

Before contacting MX Support please refer to our documentation on the MX Docs Site.

What can I expect?

Who is TechOps?

TechOps, or Technical Operations, encompasses the MX support team and our teams that support them including our Platform Support Team, the Alerting and Monitoring team, our Integration Engineering Team, and our Continuous Delivery and Operations team. 

What can I expect from TechOps?

The MX TechOps team has years of experience handling complicated technical questions and unique user scenarios. 

End-users (if applicable) will be able to reach the team by submitting in-app requests. Clients are able to submit requests via the MX Support Portal.

If you have an issue with platform availability in the production environment, please refer to the MX Status Page and/or use the critical support line. Partners and clients can get the critical support number from your account representative. Both are available 24/7/365.

Hours of Operation

• Monday - Friday 9am to 5 pm MT
• Closed during most U.S Federal Holidays
• If applicable, a critical support line is available 24 hours a day and 7 days a week

How we measure our success

These are the metrics that we use to measure ourselves. 

First Reply Time

The average time from when a ticket was created, to the time the first MX team member makes the first public comment on the ticket. 

Ticket Age

The average age of all of our current tickets that are not in a solved or closed status.

Resolution Time

The average time from when you create your ticket to our team resolving it is our resolution time.

Satisfaction Scoring (CSAT)

48 hours after one of our team members resolves your ticket, you will receive a survey to fill out about your experience. We look at the number of satisfied (good) responses, divided by the number of unsatisfied (bad) responses to compute our customer satisfaction score. 

Types of MX Support 

Support and Your Customers

When it comes to how your users are supported with the MX platform, you are in control. We offer two solutions to help you, help them.

MX provided technical support

End users communicate directly with MX Team Members.

In this scenario, end users will submit their tickets directly to our ticket management system from inside the MX experience. Then the support team will work directly with them via email until the issue is resolved.

Client provided support with escalation support from MX 

End users communicate with you, then you communicate with MX team members.

In this scenario we will hide the in-app support option, and you will handle all front end communications with your end users. If you get stuck, the MX support team is always here to help via the MX Support Portal.

How to Submit a Ticket 

Use the following articles to learn how to set up your own support account, navigate the portal, submit a new ticket, view your ticket status and history, and use our best practices when submitting a ticket.

• How To Submit A Support Ticket To MX Support - Video Link 
• Submitting a Ticket-Best Practices

Please note that there are controls within your organization’s view in Zendesk. These allow everyone in an organization to view all the organization’s tickets and add comments versus team members being able to view and add comments to only their own tickets. By default we will set it to everyone in your organization being able to view all organization tickets and add comments to them. Please let us know if you would like that to be set up differently.

Ticket Statuses

Once you create and submit a ticket you will receive an automated email after submission so you can be sure that your ticket was received by MX. 

You will receive an initial response within approximately 24 business hours from a team member. That team member will be dedicated to supporting you and will provide regular updates until your ticket has been solved. 

Your ticket will move between statuses from open, to on-hold, to pending throughout the lifecycle of the ticket. Please see below for more information on ticket statuses.

Status Definitions

New

Ticket has been created by a user and an automatic response back to the user has been sent with the content of what they submitted.

Open

The issue has been assigned to a team member who is actively working to resolve it. First response time is based on moving from "new” to "open" status.

Pending

The assigned team member has a follow-up question and/or may need more information from the user. Requests that are set to pending will alert the user via email that the ticket will be marked as solved after a maximum of 7 business days if there is no response. 

On-hold

The issue is awaiting resolution from an internal engineer or a third party. When the team member has an update they will reach out in the case, the issue will be updated and the user will be notified.

Solved

The issue has been deemed solved by the team member, or has closed due to no reply from the customer when the ticket is in pending status, and will officially close at maximum after 5 business days. The ticket can be reopened during this time by replying to any of the emails or be reopened within Zendesk

Closed

The ticket is complete. The user can respond to the email if the issue is not resolved to create a follow up ticket and reference the original ticket number.

Escalating Your Ticket 

If you feel that no progress has been made to remediate your issue, you may escalate to the MX support escalation team by clicking on the “Request Ticket Escalation” section in the main client ticket submission page of Zendesk. Please also refer to the priority definitions below.

• P1 (URGENT): Complete system down on the MX side
  • For P1 issues it is highly recommended that you use the Critical Support Line to escalate.
• P2 (HIGH) - Complete outage for a single institution or client, larger-scale issue, a request from a VIP, an outage, the platform is down, or an issue that is impacting multiple users.
  • No response or progress after a business day
• P3 (NORMAL) - Standard support request 
  • No response or progress after 3 business days

Additionally, you may contact your account representative and they will follow the appropriate steps to advocate and escalate for you internally.

Maintenance and Downtime

Please refer to the MX Status Page for information on Maintenance and Downtime 

How MX communicates maintenance and downtime

MX will occasionally need to perform maintenance on MX products to keep them running smoothly. We try to perform these updates during off-peak hours, with as much advanced notification as we can, to reduce customer impact as much as possible.

If MX needs to perform maintenance we will send you an initial notification at least 30 days in advance. 

This notification will include:

• What type of maintenance is being performed
• When the maintenance is occuring
• Which environment is being impacted (INT or PROD)
• What products will be impacted
• How long the maintenance window will last

The event, and details surrounding the changes that will be made, will also be listed on status.mx.com. We will let you know all of the details around the event including any action that might be necessary from your end.

Following the initial notification, we will send you reminders throughout the month. Reminders will be sent up to a few hours before the maintenance event starts to keep the event front of mind.

When the maintenance event starts, MX will post a maintenance update to the status page (status.mx.com). Notifications will go out to those subscribed to receive them.

Once the maintenance is complete,  we will provide that update on the status page. This update will include any additional information or action needed.

Unscheduled downtime

If there is an unexpected downtime event or platform outage, MX will update the status page to communicate any platform impacting issues. “Platform impacting” means an application, or set of applications, is negatively impacting multiple clients. The status page will provide any updates MX is taking to resolve the platform impacting issue for any of MX’s front-facing client applications. Note that all updates will be sent to those who have elected to receive them via an opt-in subscription option.

Status Page Updates Summary

The expectations for status page updates will be as follows:

1. The initial communication of an issue will be posted with an “INVESTIGATING” status label.
2. Next, one of two things will happen. 
   • MX has identified the problem and will mark the incident with the  “IDENTIFIED” status label.
   • MX will update the incident letting clients know that we are still investigating. The label will be set to “INVESTIGATING”.
3. The incident status will be set to “MITIGATED” once the problem has been resolved, meaning the errors have stopped and traffic has normalized. We will continue to monitor for the next hour to ensure the problem has not returned.
4. The incident status will be changed to “RESOLVED” if the issue does not recur.
5. Post resolution, MX will conduct a root cause analysis. If applicable, we will send out the full details of the incident in a formal RCA (root cause analysis) document.

Contacting MX

Incident Identification and Communication

MX observability teams proactively identify platform-wide impacting issues and will promptly post to the MX status page if found. If your MX service(s) appear to be impacted, please visit the status page to confirm whether or not MX has already identified the issue and is working toward resolution. No support contact is needed in that case. However, if our services are reporting “OPERATIONAL” status, please follow the “Ticket Submission” process below.

If you are experiencing an issue that is considered Critical or High severity (see Severity Level Metrics table below), submit a support ticket and, if appropriate, contact the critical support line, available 24/7/365. If your issue falls into the Medium or Low severity levels, submit a support ticket and we will handle your items during standard support hours.

Identifying and Reporting Client-Specific Issues

Client-specific issues refer to those only impacting a particular client’s user-base versus something more widespread. The proposed support path would be the following:

1. Visit the MX status page and review the current status for the impacted service
2. Proceed based on the reported status
   • An “OPERATIONAL” status suggests a client-specific issue. Reach out according to the “Contact Methodology” table below
   • An “INVESTIGATING” status suggests a widespread issue that’s already actively being addressed. No support contact is needed.
3. If the reported issue worsens, or isn’t addressed in a timely manner, a follow up in the ticket is advised.

Ticket Submission

A client can call the Critical Support line at any time. The client will ensure that before calling have created a ticket and included the following:

1. A brief description of the impact the client is experiencing
2. The % of requests being impacted
3. Timestamp on when the problem started

This will ensure that our Support Escalation team has all the necessary information to escalate internally to the respective engineering team.

Again, any updates on the case will be updated on the ticket the client submitted. Re-calling the Critical Support line for updates is discouraged. If a client would like an update on the incident they are facing they can respond to the ticket opened with MX to receive additional updates. 

Once the issue has been resolved MX will respond to the ticket validating the issue has been resolved before closing out the ticket. Once mitigation and resolution have been verified, the ticket will then be closed. 

Contact Methodology

Incident Status Definitions

Severity Level Metrics

Severity 1

Critical > 90% (Major Incident)

This severity level represents a complete system outage where all users are affected. This means the services that are in the production environment are not available and there is no acceptable workaround or alternative solution readily available. This level of an issue will manifest itself as an error in the interface. All users are affected.

Severity 2
High 50% - 89% (Major Incident*)

An issue at this level will result in "maintenance mode" messaging or an error in the interface that reads "Oops something went wrong". This means that the services in the production environment are seriously affected and operationally limited and there is no acceptable workaround or alternative solution readily available. It may also indicate a high profile user has an issue, or a time sensitive issue. Most users are affected.

Severity 3

Medium 10% - 49% Moderate Incident

This is the severity level that is the default priority for tickets or any standard request. The services in the production environment are restricted but operational, and no acceptable workaround or alternative solution is readily available. Some users are affected.

Severity 4

Low <10% Minor Incident

These are feature requests, individual transaction classification, or categorization items. This also represents questions or comments about the MX platform or services. When this severity category is assigned the services in the production environment are generally operational and there is an alternate solution available.

*MX uses the “Major Incident” label for incidents that are both Severity 1 (Critical) and Severity 2 (High).

Outages vs. Latency

The difference between outages and latency is as follows:

• Outage = Requests returning errors, exceeding service thresholds
• Latency = Requests taking longer to complete than expected

Although there are some instances when latencies would report errors, and be considered a “failure”, many occurrences of latency don’t produce errors. These definitions are consistent with the behavior that a client may experience with the services they use.

Incident-Resolution Process

Pre-Resolution

As we’ve stated previously, MX observability teams proactively identify platform-wide impacting issues and will post to the MX status page. In rare cases, clients may contact MX about a critical issue impacting their service or the platform, prompting an update to the status page. In both cases, the MX Customer Support team will partner closely with MX engineers to promptly investigate and mitigate the issue.

Post Resolution 

After the issue has been resolved, and/or any ticket submitted to our Support Escalation team has been closed, we will transition to a post resolution stage.  MX will immediately initiate a “postmortem” for the platform impacting event. During this time, clients may reach out to their Account Executive to be provided with an Incident Report.

Postmortem

A postmortem is a deep-dive exercise for the platform impacting event. All factors will be comprehensively scrutinized and analyzed to determine the root cause, learnings captured, and action items assigned to prevent future recurrence. Once the postmortem has concluded, an internal report is reviewed by the Support Escalation team, which in turn becomes an Incident Report that can be reviewed by our customers.

Incident report

An Incident Report (a.k.a Root Cause Analysis (RCA) report). Contains a detailed, high-level report with the following information:

• Description - A general overview of what happened and any symptoms that occurred during the incident. This will include a timestamp when the problem occurred. This section will also contain the root cause of the incident. 
• Duration - The duration will include a Start date/time to end date/time with an overall time (in minutes). If the timeframe is over 2 hours, the format for the total will be in the following format: {# hours, # minutes}. If the difference between the incident and resolution window is greater than 24 hours, it will be formatted as follows: {# days, # hours, # minutes}.
• Resolution - The steps taken to resolve the incident. 
• Impact - What product and/or Services within MX were affected by the incident. This will include 4 levels of severity.
• Mitigation - If applicable, this communicates any additional actions or items needed to prevent the issue from recurring. 

Delivering an Incident Report

If a client has requested an Incident report, they can expect their Account Executive to deliver it no later than 14 business days. MX strives to deliver these reports much sooner than this time window. This is typically delivered via the email that the client’s Account Executive uses to communicate with each client.

Integration Support

When you need integration support, please reach out to MX Support through a Zendesk ticket. The support team will review your request and if necessary bring in the Tech Ops Integrations team to assist. 

Ticket Reporting

Please contact your account representative if you would like to receive support ticket reporting and they can assist with this.